Doc/review api ref storage
This commit is contained in:
Kirill Chalov
committed by
Krzysztof Budzynski
Krzysztof Budzynski
parent
e20b37aff9
commit
4faf2de035
@@ -4,54 +4,60 @@ Non-volatile storage library
|
||||
Introduction
|
||||
------------
|
||||
|
||||
Non-volatile storage (NVS) library is designed to store key-value pairs in flash. This sections introduces some concepts used by NVS.
|
||||
Non-volatile storage (NVS) library is designed to store key-value pairs in flash. This section introduces some concepts used by NVS.
|
||||
|
||||
|
||||
Underlying storage
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Currently NVS uses a portion of main flash memory through ``spi_flash_{read|write|erase}`` APIs. The library uses the all the partitions with ``data`` type and ``nvs`` subtype. The application can choose to use the partition with label ``nvs`` through ``nvs_open`` API or any of the other partition by specifying its name through ``nvs_open_from_part`` API.
|
||||
Currently, NVS uses a portion of main flash memory through ``spi_flash_{read|write|erase}`` APIs. The library uses all the partitions with ``data`` type and ``nvs`` subtype. The application can choose to use the partition with the label ``nvs`` through the ``nvs_open`` API function or any other partition by specifying its name using the ``nvs_open_from_part`` API function.
|
||||
|
||||
Future versions of this library may add other storage backends to keep data in another flash chip (SPI or I2C), RTC, FRAM, etc.
|
||||
Future versions of this library may have other storage backends to keep data in another flash chip (SPI or I2C), RTC, FRAM, etc.
|
||||
|
||||
.. note:: if an NVS partition is truncated (for example, when the partition table layout is changed), its contents should be erased. ESP-IDF build system provides a ``make erase_flash`` target to erase all contents of the flash chip.
|
||||
|
||||
.. note:: NVS works best for storing many small values, rather than a few large values of type 'string' and 'blob'. If storing large blobs or strings is required, consider using the facilities provided by the FAT filesystem on top of the wear levelling library.
|
||||
.. note:: NVS works best for storing many small values, rather than a few large values of the type 'string' and 'blob'. If you need to store large blobs or strings, consider using the facilities provided by the FAT filesystem on top of the wear levelling library.
|
||||
|
||||
|
||||
Keys and values
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
NVS operates on key-value pairs. Keys are ASCII strings, maximum key length is currently 15 characters. Values can have one of the following types:
|
||||
NVS operates on key-value pairs. Keys are ASCII strings; the maximum key length is currently 15 characters. Values can have one of the following types:
|
||||
|
||||
- integer types: ``uint8_t``, ``int8_t``, ``uint16_t``, ``int16_t``, ``uint32_t``, ``int32_t``, ``uint64_t``, ``int64_t``
|
||||
- zero-terminated string
|
||||
- variable length binary data (blob)
|
||||
|
||||
.. note::
|
||||
String values are currently limited to 4000 bytes. This includes the null terminator. Blob values are limited to 508000 bytes or (97.6% of the partition size - 4000) bytes whichever is lower.
|
||||
|
||||
Additional types, such as ``float`` and ``double`` may be added later.
|
||||
String values are currently limited to 4000 bytes. This includes the null terminator. Blob values are limited to 508000 bytes or 97.6% of the partition size - 4000 bytes, whichever is lower.
|
||||
|
||||
Keys are required to be unique. Writing a value for a key which already exists behaves as follows:
|
||||
Additional types, such as ``float`` and ``double`` might be added later.
|
||||
|
||||
- if the new value is of the same type as old one, value is updated
|
||||
- if the new value has different data type, an error is returned
|
||||
Keys are required to be unique. Assigning a new value to an existing key works as follows:
|
||||
|
||||
- if the new value is of the same type as the old one, value is updated
|
||||
- if the new value has a different data type, an error is returned
|
||||
|
||||
Data type check is also performed when reading a value. An error is returned if the data type of the read operation does not match the data type of the value.
|
||||
|
||||
Data type check is also performed when reading a value. An error is returned if data type of read operation doesn’t match the data type of the value.
|
||||
|
||||
Namespaces
|
||||
^^^^^^^^^^
|
||||
|
||||
To mitigate potential conflicts in key names between different components, NVS assigns each key-value pair to one of namespaces. Namespace names follow the same rules as key names, i.e. 15 character maximum length. Namespace name is specified in the ``nvs_open`` or ``nvs_open_from_part`` call. This call returns an opaque handle, which is used in subsequent calls to ``nvs_read_*``, ``nvs_write_*``, and ``nvs_commit`` functions. This way, handle is associated with a namespace, and key names will not collide with same names in other namespaces.
|
||||
Please note that the namespaces with same name in different NVS partitions are considered as separate namespaces.
|
||||
To mitigate potential conflicts in key names between different components, NVS assigns each key-value pair to one of namespaces. Namespace names follow the same rules as key names, i.e., the maximum length is 15 characters. Namespace name is specified in the ``nvs_open`` or ``nvs_open_from_part`` call. This call returns an opaque handle, which is used in subsequent calls to the ``nvs_read_*``, ``nvs_write_*``, and ``nvs_commit`` functions. This way, a handle is associated with a namespace, and key names will not collide with same names in other namespaces.
|
||||
Please note that the namespaces with the same name in different NVS partitions are considered as separate namespaces.
|
||||
|
||||
|
||||
Security, tampering, and robustness
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
NVS is not directly compatible with the ESP32 flash encryption system. However, data can still be stored in encrypted form if NVS encryption is used together with ESP32 flash encryption. Please refer to :ref:`nvs_encryption` for more details.
|
||||
|
||||
If NVS encryption is not used, it is possible for anyone with physical access to the flash chip to alter, erase, or add key-value pairs. With NVS encryption enabled, it is not possible to alter or add a key-value pair and get recognized as a valid pair without knowing corresponding NVS encryption keys. However, there is no tamper-resistance against erase operation.
|
||||
If NVS encryption is not used, it is possible for anyone with physical access to the flash chip to alter, erase, or add key-value pairs. With NVS encryption enabled, it is not possible to alter or add a key-value pair and get recognized as a valid pair without knowing corresponding NVS encryption keys. However, there is no tamper-resistance against the erase operation.
|
||||
|
||||
The library does try to recover from conditions when flash memory is in an inconsistent state. In particular, one should be able to power off the device at any point and time and then power it back on. This should not result in loss of data, except for the new key-value pair if it was being written at the moment of powering off. The library should also be able to initialize properly with any random data present in flash memory.
|
||||
|
||||
The library does try to recover from conditions when flash memory is in an inconsistent state. In particular, one should be able to power off the device at any point and time and then power it back on. This should not result in loss of data, except for the new key-value pair if it was being written at the moment of power off. The library should also be able to initialize properly with any random data present in flash memory.
|
||||
|
||||
Internals
|
||||
---------
|
||||
@@ -59,7 +65,7 @@ Internals
|
||||
Log of key-value pairs
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
NVS stores key-value pairs sequentially, with new key-value pairs being added at the end. When a value of any given key has to be updated, new key-value pair is added at the end of the log and old key-value pair is marked as erased.
|
||||
NVS stores key-value pairs sequentially, with new key-value pairs being added at the end. When a value of any given key has to be updated, a new key-value pair is added at the end of the log and the old key-value pair is marked as erased.
|
||||
|
||||
Pages and entries
|
||||
^^^^^^^^^^^^^^^^^
|
||||
@@ -67,22 +73,22 @@ Pages and entries
|
||||
NVS library uses two main entities in its operation: pages and entries. Page is a logical structure which stores a portion of the overall log. Logical page corresponds to one physical sector of flash memory. Pages which are in use have a *sequence number* associated with them. Sequence numbers impose an ordering on pages. Higher sequence numbers correspond to pages which were created later. Each page can be in one of the following states:
|
||||
|
||||
Empty/uninitialized
|
||||
Flash storage for the page is empty (all bytes are ``0xff``). Page isn't used to store any data at this point and doesn’t have a sequence number.
|
||||
Flash storage for the page is empty (all bytes are ``0xff``). Page is not used to store any data at this point and does not have a sequence number.
|
||||
|
||||
Active
|
||||
Flash storage is initialized, page header has been written to flash, page has a valid sequence number. Page has some empty entries and data can be written there. At most one page can be in this state at any given moment.
|
||||
Flash storage is initialized, page header has been written to flash, page has a valid sequence number. Page has some empty entries and data can be written there. No more than one page can be in this state at any given moment.
|
||||
|
||||
Full
|
||||
Flash storage is in a consistent state and is filled with key-value pairs.
|
||||
Writing new key-value pairs into this page is not possible. It is still possible to mark some key-value pairs as erased.
|
||||
|
||||
Erasing
|
||||
Non-erased key-value pairs are being moved into another page so that the current page can be erased. This is a transient state, i.e. page should never stay in this state when any API call returns. In case of a sudden power off, move-and-erase process will be completed upon next power on.
|
||||
Non-erased key-value pairs are being moved into another page so that the current page can be erased. This is a transient state, i.e., page should never stay in this state at the time when any API call returns. In case of a sudden power off, the move-and-erase process will be completed upon the next power-on.
|
||||
|
||||
Corrupted
|
||||
Page header contains invalid data, and further parsing of page data was canceled. Any items previously written into this page will not be accessible. Corresponding flash sector will not be erased immediately, and will be kept along with sectors in *uninitialized* state for later use. This may be useful for debugging.
|
||||
Page header contains invalid data, and further parsing of page data was canceled. Any items previously written into this page will not be accessible. The corresponding flash sector will not be erased immediately and will be kept along with sectors in *uninitialized* state for later use. This may be useful for debugging.
|
||||
|
||||
Mapping from flash sectors to logical pages doesn't have any particular order. Library will inspect sequence numbers of pages found in each flash sector and organize pages in a list based on these numbers.
|
||||
Mapping from flash sectors to logical pages does not have any particular order. The library will inspect sequence numbers of pages found in each flash sector and organize pages in a list based on these numbers.
|
||||
|
||||
::
|
||||
|
||||
@@ -101,11 +107,11 @@ Mapping from flash sectors to logical pages doesn't have any particular order. L
|
||||
Structure of a page
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
For now we assume that flash sector size is 4096 bytes and that ESP32 flash encryption hardware operates on 32-byte blocks. It is possible to introduce some settings configurable at compile-time (e.g. via menuconfig) to accommodate flash chips with different sector sizes (although it is not clear if other components in the system, e.g. SPI flash driver and SPI flash cache can support these other sizes).
|
||||
For now, we assume that flash sector size is 4096 bytes and that ESP32 flash encryption hardware operates on 32-byte blocks. It is possible to introduce some settings configurable at compile-time (e.g., via menuconfig) to accommodate flash chips with different sector sizes (although it is not clear if other components in the system, e.g., SPI flash driver and SPI flash cache can support these other sizes).
|
||||
|
||||
Page consists of three parts: header, entry state bitmap, and entries themselves. To be compatible with ESP32 flash encryption, entry size is 32 bytes. For integer types, entry holds one key-value pair. For strings and blobs, an entry holds part of key-value pair (more on that in the entry structure description).
|
||||
|
||||
The following diagram illustrates page structure. Numbers in parentheses indicate size of each part in bytes. ::
|
||||
The following diagram illustrates the page structure. Numbers in parentheses indicate the size of each part in bytes. ::
|
||||
|
||||
+-----------+--------------+-------------+-------------------------+
|
||||
| State (4) | Seq. no. (4) | version (1) | Unused (19) | CRC32 (4) | Header (32)
|
||||
@@ -122,23 +128,23 @@ The following diagram illustrates page structure. Numbers in parentheses indicat
|
||||
| Entry 125 (32) |
|
||||
+------------------------------------------------------------------+
|
||||
|
||||
Page header and entry state bitmap are always written to flash unencrypted. Entries are encrypted if flash encryption feature of the ESP32 is used.
|
||||
Page header and entry state bitmap are always written to flash unencrypted. Entries are encrypted if flash encryption feature of ESP32 is used.
|
||||
|
||||
Page state values are defined in such a way that changing state is possible by writing 0 into some of the bits. Therefore it not necessary to erase the page to change page state, unless that is a change to *erased* state.
|
||||
Page state values are defined in such a way that changing state is possible by writing 0 into some of the bits. Therefore it is not necessary to erase the page to change its state unless that is a change to the *erased* state.
|
||||
|
||||
The version field in the header reflects NVS format version used. For backward compatibility reasons, it is decremented for every version upgrade starting at 0xff (i.e. 0xff for version-1, 0xfe for version-2 and so on).
|
||||
The version field in the header reflects the NVS format version used. For backward compatibility reasons, it is decremented for every version upgrade starting at 0xff (i.e., 0xff for version-1, 0xfe for version-2 and so on).
|
||||
|
||||
CRC32 value in header is calculated over the part which doesn't include state value (bytes 4 to 28). Unused part is currently filled with ``0xff`` bytes.
|
||||
CRC32 value in the header is calculated over the part which does not include a state value (bytes 4 to 28). The unused part is currently filled with ``0xff`` bytes.
|
||||
|
||||
The following sections describe structure of entry state bitmap and entry itself.
|
||||
The following sections describe the structure of entry state bitmap and entry itself.
|
||||
|
||||
Entry and entry state bitmap
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Each entry can be in one of the following three states. Each state is represented with two bits in the entry state bitmap. Final four bits in the bitmap (256 - 2 * 126) are unused.
|
||||
Each entry can be in one of the following three states represented with two bits in the entry state bitmap. The final four bits in the bitmap (256 - 2 * 126) are not used.
|
||||
|
||||
Empty (2'b11)
|
||||
Nothing is written into the specific entry yet. It is in an uninitialized state (all bytes ``0xff``).
|
||||
Nothing is written into the specific entry yet. It is in an uninitialized state (all bytes are ``0xff``).
|
||||
|
||||
Written (2'b10)
|
||||
A key-value pair (or part of key-value pair which spans multiple entries) has been written into the entry.
|
||||
@@ -152,7 +158,7 @@ Erased (2'b00)
|
||||
Structure of entry
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
For values of primitive types (currently integers from 1 to 8 bytes long), entry holds one key-value pair. For string and blob types, entry holds part of the whole key-value pair. For strings, in case when a key-value pair spans multiple entries, all entries are stored in the same page. Blobs are allowed to span over multiple pages by dividing them into smaller chunks. For the purpose tracking these chunks, an additional fixed length metadata entry is stored called "blob index" entry. Earlier format of blobs are still supported (can be read and modified). However, once the blobs are modified, they are stored using the new format.
|
||||
For values of primitive types (currently integers from 1 to 8 bytes long), entry holds one key-value pair. For string and blob types, entry holds part of the whole key-value pair. For strings, in case when a key-value pair spans multiple entries, all entries are stored in the same page. Blobs are allowed to span over multiple pages by dividing them into smaller chunks. For tracking these chunks, an additional fixed length metadata entry is stored called "blob index". Earlier formats of blobs are still supported (can be read and modified). However, once the blobs are modified, they are stored using the new format.
|
||||
|
||||
::
|
||||
|
||||
@@ -160,10 +166,10 @@ For values of primitive types (currently integers from 1 to 8 bytes long), entry
|
||||
| NS (1) | Type (1) | Span (1) | ChunkIndex (1) | CRC32 (4) | Key (16) | Data (8) |
|
||||
+--------+----------+----------+----------------+-----------+---------------+----------+
|
||||
|
||||
Primitive +--------------------------------+
|
||||
+--------> | Data (8) |
|
||||
Primitive +--------------------------------+
|
||||
+--------> | Data (8) |
|
||||
| Types +--------------------------------+
|
||||
+-> Fixed length --
|
||||
+-> Fixed length --
|
||||
| | +---------+--------------+---------------+-------+
|
||||
| +--------> | Size(4) | ChunkCount(1)| ChunkStart(1) | Rsv(2)|
|
||||
Data format ---+ Blob Index +---------+--------------+---------------+-------+
|
||||
@@ -176,25 +182,25 @@ For values of primitive types (currently integers from 1 to 8 bytes long), entry
|
||||
Individual fields in entry structure have the following meanings:
|
||||
|
||||
NS
|
||||
Namespace index for this entry. See section on namespaces implementation for explanation of this value.
|
||||
Namespace index for this entry. For more information on this value, see the section on namespaces implementation.
|
||||
|
||||
Type
|
||||
One byte indicating data type of value. See ``ItemType`` enumeration in ``nvs_types.h`` for possible values.
|
||||
One byte indicating the value data type. See the ``ItemType`` enumeration in ``nvs_types.h`` for possible values.
|
||||
|
||||
Span
|
||||
Number of entries used by this key-value pair. For integer types, this is equal to 1. For strings and blobs this depends on value length.
|
||||
Number of entries used by this key-value pair. For integer types, this is equal to 1. For strings and blobs, this depends on value length.
|
||||
|
||||
ChunkIndex
|
||||
Used to store index of the blob-data chunk for blob types. For other types, this should be ``0xff``.
|
||||
Used to store the index of a blob-data chunk for blob types. For other types, this should be ``0xff``.
|
||||
|
||||
CRC32
|
||||
Checksum calculated over all the bytes in this entry, except for the CRC32 field itself.
|
||||
|
||||
Key
|
||||
Zero-terminated ASCII string containing key name. Maximum string length is 15 bytes, excluding zero terminator.
|
||||
Zero-terminated ASCII string containing a key name. Maximum string length is 15 bytes, excluding a zero terminator.
|
||||
|
||||
Data
|
||||
For integer types, this field contains the value itself. If the value itself is shorter than 8 bytes it is padded to the right, with unused bytes filled with ``0xff``.
|
||||
For integer types, this field contains the value itself. If the value itself is shorter than 8 bytes, it is padded to the right, with unused bytes filled with ``0xff``.
|
||||
|
||||
For "blob index" entry, these 8 bytes hold the following information about data-chunks:
|
||||
|
||||
@@ -205,23 +211,23 @@ Data
|
||||
(Only for blob index.) Total number of blob-data chunks into which the blob was divided during storage.
|
||||
|
||||
- ChunkStart
|
||||
(Only for blob index.) ChunkIndex of the first blob-data chunk of this blob. Subsequent chunks have chunkIndex incrementally allocated (step of 1)
|
||||
(Only for blob index.) ChunkIndex of the first blob-data chunk of this blob. Subsequent chunks have chunkIndex incrementally allocated (step of 1).
|
||||
|
||||
For string and blob data chunks, these 8 bytes hold additional data about the value, described next:
|
||||
For string and blob data chunks, these 8 bytes hold additional data about the value, which are described below:
|
||||
|
||||
- Size
|
||||
(Only for strings and blobs.) Size, in bytes, of actual data. For strings, this includes zero terminator.
|
||||
(Only for strings and blobs.) Size, in bytes, of actual data. For strings, this includes zero terminators.
|
||||
|
||||
- CRC32
|
||||
(Only for strings and blobs.) Checksum calculated over all bytes of data.
|
||||
|
||||
Variable length values (strings and blobs) are written into subsequent entries, 32 bytes per entry. `Span` field of the first entry indicates how many entries are used.
|
||||
Variable length values (strings and blobs) are written into subsequent entries, 32 bytes per entry. The `Span` field of the first entry indicates how many entries are used.
|
||||
|
||||
|
||||
Namespaces
|
||||
^^^^^^^^^^
|
||||
|
||||
As mentioned above, each key-value pair belongs to one of the namespaces. Namespaces identifiers (strings) are stored as keys of key-value pairs in namespace with index 0. Values corresponding to these keys are indexes of these namespaces.
|
||||
As mentioned above, each key-value pair belongs to one of the namespaces. Namespace identifiers (strings) are stored as keys of key-value pairs in namespace with index 0. Values corresponding to these keys are indexes of these namespaces.
|
||||
|
||||
::
|
||||
|
||||
@@ -239,23 +245,23 @@ As mentioned above, each key-value pair belongs to one of the namespaces. Namesp
|
||||
Item hash list
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
To reduce the number of reads performed from flash memory, each member of Page class maintains a list of pairs: (item index; item hash). This list makes searches much quicker. Instead of iterating over all entries, reading them from flash one at a time, ``Page::findItem`` first performs search for item hash in the hash list. This gives the item index within the page, if such an item exists. Due to a hash collision it is possible that a different item will be found. This is handled by falling back to iteration over items in flash.
|
||||
To reduce the number of reads from flash memory, each member of the Page class maintains a list of pairs: item index; item hash. This list makes searches much quicker. Instead of iterating over all entries, reading them from flash one at a time, ``Page::findItem`` first performs a search for the item hash in the hash list. This gives the item index within the page if such an item exists. Due to a hash collision, it is possible that a different item will be found. This is handled by falling back to iteration over items in flash.
|
||||
|
||||
Each node in hash list contains a 24-bit hash and 8-bit item index. Hash is calculated based on item namespace, key name and ChunkIndex. CRC32 is used for calculation, result is truncated to 24 bits. To reduce overhead of storing 32-bit entries in a linked list, list is implemented as a doubly-linked list of arrays. Each array holds 29 entries, for the total size of 128 bytes, together with linked list pointers and 32-bit count field. Minimal amount of extra RAM usage per page is therefore 128 bytes, maximum is 640 bytes.
|
||||
Each node in the hash list contains a 24-bit hash and 8-bit item index. Hash is calculated based on item namespace, key name, and ChunkIndex. CRC32 is used for calculation; the result is truncated to 24 bits. To reduce the overhead for storing 32-bit entries in a linked list, the list is implemented as a double-linked list of arrays. Each array holds 29 entries, for the total size of 128 bytes, together with linked list pointers and a 32-bit count field. The minimum amount of extra RAM usage per page is therefore 128 bytes; maximum is 640 bytes.
|
||||
|
||||
.. _nvs_encryption:
|
||||
|
||||
NVS Encryption
|
||||
--------------
|
||||
|
||||
Data stored in NVS partitions can be encrypted using AES-XTS in the manner similar to one mentioned in disc encryption standard IEEE P1619. For the purpose of encryption, each entry is considered as one `sector` and relative address of the entry (w.r.t. partition-start) is fed to the encryption algorithm as `sector-number`. The keys required for nvs encryption are stored in yet another partition, which is protected using :doc:`Flash Encryption <../../security/flash-encryption>`. Therefore, enabling :doc:`Flash Encryption <../../security/flash-encryption>` is a prerequisite for NVS encryption.
|
||||
Data stored in NVS partitions can be encrypted using AES-XTS in the manner similar to the one mentioned in disk encryption standard IEEE P1619. For the purpose of encryption, each entry is treated as one `sector` and relative address of the entry (w.r.t. partition-start) is fed to the encryption algorithm as `sector-number`. The keys required for NVS encryption are stored in yet another partition, which is protected using :doc:`Flash Encryption <../../security/flash-encryption>`. Therefore, enabling :doc:`Flash Encryption <../../security/flash-encryption>` is a prerequisite for NVS encryption.
|
||||
|
||||
.. _nvs_key_partition:
|
||||
|
||||
NVS key partition
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
An application requiring NVS encryption support needs to be compiled with a key-partition of type `data` and subtype `key`. This partition should be marked as `encrypted`. Refer to :doc:`Partition Tables <../../api-guides/partition-tables>` for more details. The size of the partition should be 4096 bytes (minimum partition size). The structure of this partition is depicted below.
|
||||
An application requiring NVS encryption support needs to be compiled with a key-partition of the type `data` and subtype `key`. This partition should be marked as `encrypted`. Refer to :doc:`Partition Tables <../../api-guides/partition-tables>` for more details. The size of the partition should be 4096 bytes (minimum partition size). The structure of this partition is depicted below.
|
||||
|
||||
::
|
||||
|
||||
@@ -267,25 +273,33 @@ An application requiring NVS encryption support needs to be compiled with a key-
|
||||
| CRC32(4) |
|
||||
+---------------------------------------------+
|
||||
|
||||
This partition can be generated using `nvs partition generator` utility and flashed onto the device. Since the partition is marked `encrypted` and :doc:`Flash Encryption <../../security/flash-encryption>` is enabled, bootloader will encrypt this partition using flash encryption key on first boot. Alternatively, the keys can be generated after startup using ``nvs_flash_generate_keys`` API provided by ``nvs_flash.h``, which will then write those keys onto the key-partition in encrypted form.
|
||||
This partition can be generated using `nvs partition generator` utility and flashed onto the device. Since the partition is marked `encrypted` and :doc:`Flash Encryption <../../security/flash-encryption>` is enabled, bootloader will encrypt this partition using flash encryption key on the first boot. Alternatively, the keys can be generated after startup using the ``nvs_flash_generate_keys`` API function provided by ``nvs_flash.h``, which will then write those keys onto the key-partition in encrypted form.
|
||||
|
||||
It is possible for an application to use different keys for different NVS partitions and thereby have multiple key-partitions. However, it is a responsibility of the application to provide correct key-partition/keys for the purpose of encryption/decryption.
|
||||
|
||||
Encrypted Read/Write
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The same NVS APIs ``nvs_read_*`` or ``nvs_write_*`` can be used for reading and writing of encrypted nvs partition as well. However, the APIs for initialising NVS partitions are different. ``nvs_flash_secure_init`` and ``nvs_flash_secure_init_partition`` are used for initialising instead of ``nvs_flash_init`` and ``nvs_flash_init_partition`` respectively. ``nvs_sec_cfg_t`` structure required for these APIs can be populated using ``nvs_flash_generate_keys`` or ``nvs_flash_read_security_cfg``.
|
||||
The same NVS API functions ``nvs_read_*`` or ``nvs_write_*`` can be used for reading of, and writing to an encrypted nvs partition as well. However, the API functions for initialising NVS partitions are different: ``nvs_flash_secure_init`` and ``nvs_flash_secure_init_partition`` instead of ``nvs_flash_init`` and ``nvs_flash_init_partition`` respectively. The ``nvs_sec_cfg_t`` structure required for these API functions can be populated using ``nvs_flash_generate_keys`` or ``nvs_flash_read_security_cfg``.
|
||||
|
||||
Applications are expected to follow the following steps in order to perform NVS read/write operations with encryption enabled.
|
||||
Applications are expected to follow the steps below in order to perform NVS read/write operations with encryption enabled.
|
||||
|
||||
1. Find key partition and NVS data partition using ``esp_partition_find*`` APIs.
|
||||
2. Populate ``nvs_sec_cfg_t`` struct using ``nvs_flash_read_security_cfg`` or ``nvs_flash_generate_keys`` APIs.
|
||||
3. Initialise NVS flash partition using ``nvs_flash_secure_init`` or ``nvs_flash_secure_init_partition`` APIs.
|
||||
4. Open a namespace using ``nvs_open`` or ``nvs_open_from_part`` APIs
|
||||
5. Perform NVS read/write operations using ``nvs_read_*`` or ``nvs_write_*``
|
||||
6. Deinitialise NVS partition using ``nvs_flash_deinit``.
|
||||
1. Find key partition and NVS data partition using ``esp_partition_find*`` API functions.
|
||||
2. Populate the ``nvs_sec_cfg_t`` struct using the ``nvs_flash_read_security_cfg`` or ``nvs_flash_generate_keys`` API functions.
|
||||
3. Initialise NVS flash partition using the ``nvs_flash_secure_init`` or ``nvs_flash_secure_init_partition`` API functions.
|
||||
4. Open a namespace using the ``nvs_open`` or ``nvs_open_from_part`` API functions.
|
||||
5. Perform NVS read/write operations using ``nvs_read_*`` or ``nvs_write_*``.
|
||||
6. Deinitialise an NVS partition using ``nvs_flash_deinit``.
|
||||
|
||||
NVS iterators
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
Iterators allow to list key-value pairs stored in NVS based on specified partition name, namespace and data type. ``nvs_entry_find`` returns an opaque handle, which is used in subsequent calls to ``nvs_entry_next`` and ``nvs_entry_info`` function. ``nvs_entry_next`` function returns iterator to the next key-value pair. If none or no other key-value pair was found for given criteria, ``nvs_entry_find`` and ``nvs_entry_next`` functions return NULL. In that case, iterator does not have to be released. Otherwise, ``nvs_release_iterator`` function has to be used, when iterator is no longer needed. Information about each key-value pair can be obtained from ``nvs_entry_info`` function.
|
||||
Iterators allow to list key-value pairs stored in NVS, based on specified partition name, namespace, and data type.
|
||||
|
||||
There are the following functions available:
|
||||
|
||||
- ``nvs_entry_find`` returns an opaque handle, which is used in subsequent calls to the ``nvs_entry_next`` and ``nvs_entry_info`` functions.
|
||||
- ``nvs_entry_next`` returns iterator to the next key-value pair.
|
||||
- ``nvs_entry_info`` returns information about each key-value pair
|
||||
|
||||
If none or no other key-value pair was found for given criteria, ``nvs_entry_find`` and ``nvs_entry_next`` return NULL. In that case, the iterator does not have to be released. If the iterator is no longer needed, you can release it by using the function ``nvs_release_iterator``.
|
||||
|
||||
@@ -4,64 +4,81 @@ NVS Partition Generator Utility
|
||||
Introduction
|
||||
------------
|
||||
|
||||
:component_file:`nvs_flash/nvs_partition_generator/nvs_partition_gen.py` utility is designed to help create a binary file, compatible with NVS architecture defined in :doc:`Non-Volatile Storage </api-reference/storage/nvs_flash>`, based on user provided key-value pairs in a CSV file.
|
||||
Utility is ideally suited for generating a binary blob, containing data specific to ODM/OEM, which can be flashed externally at the time of device manufacturing. This helps manufacturers set unique value for various parameters for each device, e.g. serial number, while using same application firmware for all devices.
|
||||
The utility :component_file:`nvs_flash/nvs_partition_generator/nvs_partition_gen.py` creates a binary file based on key-value pairs provided in a CSV file. The binary file is compatible with NVS architecture defined in :doc:`Non-Volatile Storage </api-reference/storage/nvs_flash>`.
|
||||
This utility is ideally suited for generating a binary blob, containing data specific to ODM/OEM, which can be flashed externally at the time of device manufacturing. This allows manufacturers to generate many instances of the same application firmware with customized parameters for each device, such as a serial number.
|
||||
|
||||
|
||||
Prerequisites
|
||||
-------------
|
||||
To use this utility in encryption mode, the following packages need to be installed:
|
||||
To use this utility in encryption mode, install the following packages:
|
||||
- cryptography package
|
||||
|
||||
These dependencies is already captured by including these packages in `requirement.txt` in top level IDF directory.
|
||||
All the required packages are included in `requirements.txt` in the root of the esp-idf directory.
|
||||
|
||||
|
||||
CSV file format
|
||||
---------------
|
||||
|
||||
Each row of the .csv file should have 4 parameters, separated by comma. Below is the description of each of these parameters:
|
||||
Each line of a .csv file should contain 4 parameters, separated by a comma. The table below provides the description for each of these parameters.
|
||||
|
||||
Key
|
||||
Key of the data. Data can later be accessed from an application via this key.
|
||||
+-----+-----------+----------------------------------------------------------------------+-----------------------------------------------------+
|
||||
| No. | Parameter | Description | Notes |
|
||||
+=====+===========+======================================================================+=====================================================+
|
||||
| 1 | Key | Key of the data. The data can be accessed later from | |
|
||||
| | | an application using this key. | |
|
||||
+-----+-----------+----------------------------------------------------------------------+-----------------------------------------------------+
|
||||
| 2 | Type | Supported values are ``file``, ``data`` and ``namespace``. | |
|
||||
+-----+-----------+----------------------------------------------------------------------+-----------------------------------------------------+
|
||||
| 3 | Encoding | Supported values are: ``u8``, ``i8``, ``u16``, ``u32``, | As of now, for the ``file`` type, |
|
||||
| | | ``i32``, ``string``, ``hex2bin``, ``base64`` and ``binary``. | only ``hex2bin``, ``base64``, ``string``, |
|
||||
| | | This specifies how actual data values are encoded in the | and ``binary`` encoding is supported. |
|
||||
| | | resulting binary file. The difference between the ``string`` | |
|
||||
| | | and ``binary`` encoding is that ``string`` data is terminated | |
|
||||
| | | with a NULL character, whereas ``binary`` data is not. | |
|
||||
+-----+-----------+----------------------------------------------------------------------+-----------------------------------------------------+
|
||||
| 4 | Value | Data value. | Encoding and Value cells for the ``namespace`` |
|
||||
| | | | field type should be empty. Encoding and Value |
|
||||
| | | | of ``namespace`` is fixed and is not configurable. |
|
||||
| | | | Any values in these cells are ignored. |
|
||||
+-----+-----------+----------------------------------------------------------------------+-----------------------------------------------------+
|
||||
|
||||
Type
|
||||
Supported values are ``file``, ``data`` and ``namespace``.
|
||||
.. note:: The first line of the CSV file should always be the column header and it is not configurable.
|
||||
|
||||
Encoding
|
||||
Supported values are: ``u8``, ``i8``, ``u16``, ``u32``, ``i32``, ``string``, ``hex2bin``, ``base64`` and ``binary``. This specifies how actual data values are encoded in the resultant binary file. Difference between ``string`` and ``binary`` encoding is that ``string`` data is terminated with a NULL character, whereas ``binary`` data is not.
|
||||
|
||||
.. note:: For ``file`` type, only ``hex2bin``, ``base64``, ``string`` and ``binary`` is supported as of now.
|
||||
|
||||
Value
|
||||
Data value.
|
||||
|
||||
.. note:: Encoding and Value cells for ``namespace`` field type should be empty. Encoding and Value of ``namespace`` is fixed and isn't configurable. Any value in these cells are ignored.
|
||||
|
||||
.. note:: First row of the CSV file should always be column header and isn't configurable.
|
||||
|
||||
Below is an example dump of such CSV file::
|
||||
Below is an example dump of such a CSV file::
|
||||
|
||||
key,type,encoding,value <-- column header
|
||||
namespace_name,namespace,, <-- First entry should be of type "namespace"
|
||||
key1,data,u8,1
|
||||
key2,file,string,/path/to/file
|
||||
|
||||
.. note:: Make sure there are no spaces before and after ',' or at the end of each line in CSV file.
|
||||
|
||||
.. note::
|
||||
|
||||
Make sure there are **no spaces**:
|
||||
- before and after ','
|
||||
- at the end of each line in a CSV file
|
||||
|
||||
|
||||
NVS Entry and Namespace association
|
||||
-----------------------------------
|
||||
|
||||
When a new namespace entry is encountered in the CSV file, each follow-up entries will be part of that namespace, until next namespace entry is found, in which case all the follow-up entries will be part of the new namespace.
|
||||
When a namespace entry is encountered in a CSV file, each following entry will be treated as part of that namespace until the next namespace entry is found. At this point, all the following entries will be treated as part of the new namespace.
|
||||
|
||||
.. note:: First entry in a CSV file should always be a ``namespace`` entry.
|
||||
|
||||
.. note:: First entry in a CSV file should always be ``namespace`` entry.
|
||||
|
||||
Multipage Blob Support
|
||||
----------------------
|
||||
|
||||
By default, binary blobs are allowed to span over multiple pages and written in the format mentioned in section :ref:`structure_of_entry`.
|
||||
If older format is intended to be used, the utility provides an option to disable this feature.
|
||||
By default, binary blobs are allowed to span over multiple pages and are written in the format mentioned in Section :ref:`structure_of_entry`.
|
||||
If you intend to use an older format, the utility provides an option to disable this feature.
|
||||
|
||||
|
||||
Encryption Support
|
||||
-------------------
|
||||
This utility allows you to create an enrypted binary file also. Encryption used is AES-XTS encryption. Refer to :ref:`nvs_encryption` for more details.
|
||||
|
||||
The NVS Partition Generator utility also allows you to create an encrypted binary file. The utility uses the AES-XTS encryption. Please refer to :ref:`nvs_encryption` for more details.
|
||||
|
||||
|
||||
Running the utility
|
||||
-------------------
|
||||
@@ -74,29 +91,32 @@ Running the utility
|
||||
[--keyfile KEYFILE] [--outdir OUTDIR]
|
||||
|
||||
|
||||
+------------------------+----------------------------------------------------------------------------------------------+
|
||||
| Arguments | Description |
|
||||
+========================+==============================================================================================+
|
||||
| --input INPUT | Path to CSV file to parse. |
|
||||
+------------------------+----------------------------------------------------------------------------------------------+
|
||||
| --output OUTPUT | Path to output generated binary file. |
|
||||
+------------------------+----------------------------------------------------------------------------------------------+
|
||||
| --size SIZE | Size of NVS Partition in bytes (must be multiple of 4096) |
|
||||
+------------------------+----------------------------------------------------------------------------------------------+
|
||||
| --version {v1,v2} | Set version. Default: v2 |
|
||||
+------------------------+----------------------------------------------------------------------------------------------+
|
||||
| --keygen {true,false} | Generate keys for encryption. |
|
||||
+------------------------+----------------------------------------------------------------------------------------------+
|
||||
| --encrypt {true,false} | Set encryption mode. Default: false |
|
||||
+------------------------+----------------------------------------------------------------------------------------------+
|
||||
| --keyfile KEYFILE | File having key for encryption (Applicable only if encryption mode is true) |
|
||||
+------------------------+----------------------------------------------------------------------------------------------+
|
||||
| --outdir OUTDIR | The output directory to store the files created (Default: current directory) |
|
||||
+------------------------+----------------------------------------------------------------------------------------------+
|
||||
+------------------------+---------------------------------------------------+-------------------+
|
||||
| Arguments | Description | Default Value |
|
||||
+========================+===================================================+===================+
|
||||
| --input INPUT | Path to a CSV file to parse. | |
|
||||
+------------------------+---------------------------------------------------+-------------------+
|
||||
| --output OUTPUT | Path to the generated binary file. | |
|
||||
+------------------------+---------------------------------------------------+-------------------+
|
||||
| --size SIZE | Size of NVS Partition in bytes | |
|
||||
| | (must be multiple of 4096). | |
|
||||
+------------------------+---------------------------------------------------+-------------------+
|
||||
| --version {v1,v2} | Set version. | v2 |
|
||||
+------------------------+---------------------------------------------------+-------------------+
|
||||
| --keygen {true,false} | Generate keys for encryption. | |
|
||||
+------------------------+---------------------------------------------------+-------------------+
|
||||
| --encrypt {true,false} | Set encryption mode. Default: false. | false |
|
||||
+------------------------+---------------------------------------------------+-------------------+
|
||||
| --keyfile KEYFILE | File containing the key for encryption | |
|
||||
| | (Applicable only if encryption mode is true). | |
|
||||
+------------------------+---------------------------------------------------+-------------------+
|
||||
| --outdir OUTDIR | The output directory to store the created files. | current directory |
|
||||
+------------------------+---------------------------------------------------+-------------------+
|
||||
|
||||
You can run this utility in two modes:
|
||||
- Default mode - Binary generated in this mode is an unencrypted binary file.
|
||||
- Encryption mode - Binary generated in this mode is an encrypted binary file.
|
||||
|
||||
- **Default mode**: You get an unencrypted binary file.
|
||||
- **Encryption mode**: You get an encrypted binary file.
|
||||
|
||||
|
||||
**In default mode:**
|
||||
@@ -109,12 +129,11 @@ You can run this utility in two modes:
|
||||
[--keygen {true,false}] [--encrypt {true,false}]
|
||||
[--keyfile KEYFILE] [--outdir OUTDIR]
|
||||
|
||||
You can run the utility using below command::
|
||||
You can run the utility using the command below::
|
||||
|
||||
python nvs_partition_gen.py --input sample.csv --output sample.bin --size 0x3000
|
||||
|
||||
|
||||
|
||||
**In encryption mode:**
|
||||
-----------------------
|
||||
|
||||
@@ -126,21 +145,21 @@ You can run the utility using below command::
|
||||
[--version {v1,v2}] [--outdir OUTDIR]
|
||||
|
||||
|
||||
You can run the utility using below commands:
|
||||
You can run the utility using one of the commands below:
|
||||
|
||||
- By enabling generation of encryption keys::
|
||||
- By enabling generation of encryption keys::
|
||||
|
||||
python nvs_partition_gen.py --input sample.csv --output sample_encrypted.bin --size 0x3000 --encrypt true --keygen true
|
||||
python nvs_partition_gen.py --input sample.csv --output sample_encrypted.bin --size 0x3000 --encrypt true --keygen true
|
||||
|
||||
- By taking encryption keys as an input file. A sample encryption keys binary file is provided with the utility::
|
||||
- By taking encryption keys as an input file. A sample binary file containing encryption keys is provided with the utility::
|
||||
|
||||
python nvs_partition_gen.py --input sample.csv --output sample_encrypted.bin --size 0x3000 --encrypt true --keyfile testdata/sample_encryption_keys.bin
|
||||
python nvs_partition_gen.py --input sample.csv --output sample_encrypted.bin --size 0x3000 --encrypt true --keyfile testdata/sample_encryption_keys.bin
|
||||
|
||||
- By enabling generation of encryption keys and storing the keys in custom filename::
|
||||
- By enabling generation of encryption keys and storing the keys in a binary file with a custom filename::
|
||||
|
||||
python nvs_partition_gen.py --input sample.csv --output sample_encrypted.bin --size 0x3000 --encrypt true --keygen true --keyfile encryption_keys_generated.bin
|
||||
python nvs_partition_gen.py --input sample.csv --output sample_encrypted.bin --size 0x3000 --encrypt true --keygen true --keyfile encryption_keys_generated.bin
|
||||
|
||||
.. note:: If `--keygen` is given with `--keyfile` argument, generated keys will be stored in `--keyfile` file. If `--keygen` argument is absent, `--keyfile` is taken as input file having key for encryption.
|
||||
.. note:: If `--keygen` is given with the `--keyfile` argument, generated keys will be stored in the `--keyfile` file. If `--keygen` argument is absent, `--keyfile` is taken as input file containing encryption keys.
|
||||
|
||||
|
||||
*To generate* **only** *encryption keys with this utility*::
|
||||
@@ -181,7 +200,7 @@ A sample CSV file is provided with the utility::
|
||||
|
||||
Caveats
|
||||
-------
|
||||
- Utility doesn't check for duplicate keys and will write data pertaining to both keys. User needs to make sure keys are distinct.
|
||||
- Once a new page is created, no data will be written in the space left in previous page. Fields in the CSV file need to be ordered in such a way so as to optimize memory.
|
||||
- Utility does not check for duplicate keys and will write data pertaining to both keys. You need to make sure that the keys are distinct.
|
||||
- Once a new page is created, no data will be written in the space left on the previous page. Fields in the CSV file need to be ordered in such a way as to optimize memory.
|
||||
- 64-bit datatype is not yet supported.
|
||||
|
||||
|
||||
@@ -1,80 +1,55 @@
|
||||
SPI Flash APIs
|
||||
==============
|
||||
SPI Flash API
|
||||
=============
|
||||
|
||||
Overview
|
||||
--------
|
||||
The spi_flash component contains APIs related to reading, writing, erasing,
|
||||
memory mapping data in the external SPI flash. It also has higher-level
|
||||
APIs which work with partitions defined in the :doc:`partition table </api-guides/partition-tables>`.
|
||||
The spi_flash component contains API functions related to reading, writing, erasing, memory mapping for data in the external SPI flash. The spi_flash component also has higher-level API functions which work with partitions defined in the :doc:`partition table </api-guides/partition-tables>`.
|
||||
|
||||
Note that all the functionality is limited to the "main" SPI flash chip,
|
||||
the same SPI flash chip from which program runs. For ``spi_flash_*`` functions,
|
||||
this is a software limitation. The underlying ROM functions which work with SPI flash
|
||||
do not have provisions for working with flash chips attached to SPI peripherals
|
||||
other than SPI0.
|
||||
Note that all the functionality is limited to the "main" SPI flash chip, the same SPI flash chip from which programs are runs. For ``spi_flash_*`` functions, this is a software limitation. The underlying ROM functions which work with SPI flash do not have provisions for working with flash chips attached to SPI peripherals other than SPI0.
|
||||
|
||||
SPI flash access APIs
|
||||
---------------------
|
||||
SPI flash access API
|
||||
--------------------
|
||||
|
||||
This is the set of APIs for working with data in flash:
|
||||
This is the set of API functions for working with data in flash:
|
||||
|
||||
- :cpp:func:`spi_flash_read` used to read data from flash to RAM
|
||||
- :cpp:func:`spi_flash_write` used to write data from RAM to flash
|
||||
- :cpp:func:`spi_flash_erase_sector` used to erase individual sectors of flash
|
||||
- :cpp:func:`spi_flash_erase_range` used to erase range of addresses in flash
|
||||
- :cpp:func:`spi_flash_read` reads data from flash to RAM
|
||||
- :cpp:func:`spi_flash_write` writes data from RAM to flash
|
||||
- :cpp:func:`spi_flash_erase_sector` erases individual sectors of flash
|
||||
- :cpp:func:`spi_flash_erase_range` erases ranges of addresses in flash
|
||||
- :cpp:func:`spi_flash_get_chip_size` returns flash chip size, in bytes, as configured in menuconfig
|
||||
|
||||
Generally, try to avoid using the raw SPI flash functions in favour of
|
||||
:ref:`partition-specific functions <flash-partition-apis>`.
|
||||
Generally, try to avoid using the raw SPI flash functions in favor of :ref:`partition-specific functions <flash-partition-apis>`.
|
||||
|
||||
SPI Flash Size
|
||||
--------------
|
||||
|
||||
The SPI flash size is configured by writing a field in the software bootloader
|
||||
image header, flashed at offset 0x1000.
|
||||
The SPI flash size is configured by writing a field in the software bootloader image header, flashed at offset 0x1000.
|
||||
|
||||
By default, the SPI flash size is detected by esptool.py when this bootloader is
|
||||
written to flash, and the header is updated with the correct
|
||||
size. Alternatively, it is possible to generate a fixed flash size by setting
|
||||
:envvar:`CONFIG_ESPTOOLPY_FLASHSIZE` in ``make menuconfig``.
|
||||
By default, the SPI flash size is detected by esptool.py when this bootloader is written to flash, and the header is updated with the correct size. Alternatively, it is possible to generate a fixed flash size by setting :envvar:`CONFIG_ESPTOOLPY_FLASHSIZE` in ``make menuconfig``.
|
||||
|
||||
If it is necessary to override the configured flash size at runtime, is is
|
||||
possible to set the ``chip_size`` member of ``g_rom_flashchip`` structure. This
|
||||
size is used by ``spi_flash_*`` functions (in both software & ROM) for bounds
|
||||
checking.
|
||||
If it is necessary to override the configured flash size at runtime, it is possible to set the ``chip_size`` member of the ``g_rom_flashchip`` structure. This size is used by ``spi_flash_*`` functions (in both software & ROM) to check the bounds.
|
||||
|
||||
Concurrency Constraints
|
||||
-----------------------
|
||||
|
||||
Because the SPI flash is also used for firmware execution (via the instruction &
|
||||
data caches), these caches must be disabled while reading/writing/erasing. This
|
||||
means that both CPUs must be running code from IRAM and only reading data from
|
||||
DRAM while flash write operations occur.
|
||||
Because the SPI flash is also used for firmware execution via the instruction & data caches, these caches must be disabled while reading/writing/erasing. This means that both CPUs must be running code from IRAM and must only be reading data from DRAM while flash write operations occur.
|
||||
|
||||
If you use the APIs documented here, then this happens automatically and
|
||||
transparently. However note that it will have some performance impact on other
|
||||
tasks in the system.
|
||||
If you use the API functions documented here, then these constraints are applied automatically and transparently. However, note that it will have some performance impact on other tasks in the system.
|
||||
|
||||
Refer to the :ref:`application memory layout <memory-layout>` documentation for
|
||||
an explanation of the differences between IRAM, DRAM and flash cache.
|
||||
For differences between IRAM, DRAM, and flash cache, please refer to the :ref:`application memory layout <memory-layout>` documentation.
|
||||
|
||||
To avoid reading flash cache accidentally, when one CPU commences a flash write
|
||||
or erase operation the other CPU is put into a blocked state and all
|
||||
non-IRAM-safe interrupts are disabled on both CPUs, until the flash operation
|
||||
completes.
|
||||
To avoid reading flash cache accidentally, when one CPU initiates a flash write or erase operation, the other CPU is put into a blocked state, and all non-IRAM-safe interrupts are disabled on both CPUs until the flash operation completes.
|
||||
|
||||
If one CPU initiates a flash write or erase operation, the other CPU is put into a blocked state to avoid reading flash cache accidentally. All interrupts not safe for IRAM are disabled on both CPUs until the flash operation completes.
|
||||
|
||||
.. _iram-safe-interrupt-handlers:
|
||||
|
||||
IRAM-Safe Interrupt Handlers
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If you have an interrupt handler that you want to execute even when a flash
|
||||
operation is in progress (for example, for low latency operations), set the
|
||||
``ESP_INTR_FLAG_IRAM`` flag when the :doc:`interrupt handler is registered
|
||||
</api-reference/system/intr_alloc>`.
|
||||
If you have an interrupt handler that you want to execute while a flash operation is in progress (for example, for low latency operations), set the ``ESP_INTR_FLAG_IRAM`` flag when the :doc:`interrupt handler is registered </api-reference/system/intr_alloc>`.
|
||||
|
||||
You must ensure all data and functions accessed by these interrupt handlers are
|
||||
located in IRAM or DRAM. This includes any functions that the handler calls.
|
||||
You must ensure that all data and functions accessed by these interrupt handlers, including the ones that handlers call, are located in IRAM or DRAM.
|
||||
|
||||
Use the ``IRAM_ATTR`` attribute for functions::
|
||||
|
||||
@@ -93,80 +68,56 @@ Use the ``DRAM_ATTR`` and ``DRAM_STR`` attributes for constant data::
|
||||
const static char *MSG = DRAM_STR("I am a string stored in RAM");
|
||||
}
|
||||
|
||||
Note that knowing which data should be marked with ``DRAM_ATTR`` can be hard,
|
||||
the compiler will sometimes recognise that a variable or expression is constant
|
||||
(even if it is not marked ``const``) and optimise it into flash, unless it is
|
||||
marked with ``DRAM_ATTR``.
|
||||
Note that knowing which data should be marked with ``DRAM_ATTR`` can be hard, the compiler will sometimes recognize that a variable or expression is constant (even if it is not marked ``const``) and optimize it into flash, unless it is marked with ``DRAM_ATTR``.
|
||||
|
||||
If a function or symbol is not correctly put into IRAM/DRAM and the interrupt
|
||||
handler reads from the flash cache during a flash operation, it will cause a
|
||||
crash due to Illegal Instruction exception (for code which should be in IRAM) or
|
||||
garbage data to be read (for constant data which should be in DRAM).
|
||||
If a function or symbol is not correctly put into IRAM/DRAM, and the interrupt handler reads from the flash cache during a flash operation, it will cause a crash due to Illegal Instruction exception (for code which should be in IRAM) or garbage data to be read (for constant data which should be in DRAM).
|
||||
|
||||
.. _flash-partition-apis:
|
||||
|
||||
Partition table APIs
|
||||
--------------------
|
||||
Partition table API
|
||||
-------------------
|
||||
|
||||
ESP-IDF projects use a partition table to maintain information about various regions of
|
||||
SPI flash memory (bootloader, various application binaries, data, filesystems).
|
||||
More information about partition tables can be found :doc:`here </api-guides/partition-tables>`.
|
||||
ESP-IDF projects use a partition table to maintain information about various regions of SPI flash memory (bootloader, various application binaries, data, filesystems). More information on partition tables can be found :doc:`here </api-guides/partition-tables>`.
|
||||
|
||||
This component provides APIs to enumerate partitions found in the partition table
|
||||
and perform operations on them. These functions are declared in ``esp_partition.h``:
|
||||
This component provides API functions to enumerate partitions found in the partition table and perform operations on them. These functions are declared in ``esp_partition.h``:
|
||||
|
||||
- :cpp:func:`esp_partition_find` used to search partition table for entries with
|
||||
specific type, returns an opaque iterator
|
||||
- :cpp:func:`esp_partition_get` returns a structure describing the partition, for the given iterator
|
||||
- :cpp:func:`esp_partition_next` advances iterator to the next partition found
|
||||
- :cpp:func:`esp_partition_iterator_release` releases iterator returned by ``esp_partition_find``
|
||||
- :cpp:func:`esp_partition_find_first` is a convenience function which returns structure
|
||||
describing the first partition found by ``esp_partition_find``
|
||||
- :cpp:func:`esp_partition_read`, :cpp:func:`esp_partition_write`, :cpp:func:`esp_partition_erase_range`
|
||||
are equivalent to :cpp:func:`spi_flash_read`, :cpp:func:`spi_flash_write`,
|
||||
:cpp:func:`spi_flash_erase_range`, but operate within partition boundaries
|
||||
- :cpp:func:`esp_partition_find` checks a partition table for entries with specific type, returns an opaque iterator.
|
||||
- :cpp:func:`esp_partition_get` returns a structure describing the partition for a given iterator.
|
||||
- :cpp:func:`esp_partition_next` shifts the iterator to the next found partition.
|
||||
- :cpp:func:`esp_partition_iterator_release` releases iterator returned by ``esp_partition_find``.
|
||||
- :cpp:func:`esp_partition_find_first` - a convenience function which returns the structure describing the first partition found by ``esp_partition_find``.
|
||||
- :cpp:func:`esp_partition_read`, :cpp:func:`esp_partition_write`, :cpp:func:`esp_partition_erase_range` are equivalent to :cpp:func:`spi_flash_read`, :cpp:func:`spi_flash_write`, :cpp:func:`spi_flash_erase_range`, but operate within partition boundaries.
|
||||
|
||||
.. note::
|
||||
Most application code should use these ``esp_partition_*`` APIs instead of lower level
|
||||
``spi_flash_*`` APIs. Partition APIs do bounds checking and calculate correct
|
||||
offsets in flash based on data stored in partition table.
|
||||
Application code should mostly use these ``esp_partition_*`` API functions instead of lower level ``spi_flash_*`` API functions. Partition table API functions do bounds checking and calculate correct offsets in flash, based on data stored in a partition table.
|
||||
|
||||
SPI Flash Encryption
|
||||
--------------------
|
||||
|
||||
It is possible to encrypt SPI flash contents, and have it transparenlty decrypted by hardware.
|
||||
It is possible to encrypt the contents of SPI flash and have it transparently decrypted by hardware.
|
||||
|
||||
Refer to the :doc:`Flash Encryption documentation </security/flash-encryption>` for more details.
|
||||
|
||||
Memory mapping APIs
|
||||
-------------------
|
||||
Memory mapping API
|
||||
------------------
|
||||
|
||||
ESP32 features memory hardware which allows regions of flash memory to be mapped
|
||||
into instruction and data address spaces. This mapping works only for read operations,
|
||||
it is not possible to modify contents of flash memory by writing to mapped memory
|
||||
region. Mapping happens in 64KB pages. Memory mapping hardware can map up to
|
||||
4 megabytes of flash into data address space, and up to 16 megabytes of flash into
|
||||
instruction address space. See the technical reference manual for more details
|
||||
about memory mapping hardware.
|
||||
ESP32 features memory hardware which allows regions of flash memory to be mapped into instruction and data address spaces. This mapping works only for read operations. It is not possible to modify contents of flash memory by writing to a mapped memory region.
|
||||
|
||||
Note that some number of 64KB pages is used to map the application
|
||||
itself into memory, so the actual number of available 64KB pages may be less.
|
||||
Mapping happens in 64KB pages. Memory mapping hardware can map up to four megabytes of flash into data address space and up to 16 megabytes of flash into instruction address space. See the technical reference manual for more details about memory mapping hardware.
|
||||
|
||||
Reading data from flash using a memory mapped region is the only way to decrypt
|
||||
contents of flash when :doc:`flash encryption </security/flash-encryption>` is enabled.
|
||||
Decryption is performed at hardware level.
|
||||
Note that some 64KB pages are used to map the application itself into memory, so the actual number of available 64KB pages may be less.
|
||||
|
||||
Memory mapping APIs are declared in ``esp_spi_flash.h`` and ``esp_partition.h``:
|
||||
Reading data from flash using a memory mapped region is the only way to decrypt contents of flash when :doc:`flash encryption </security/flash-encryption>` is enabled. Decryption is performed at the hardware level.
|
||||
|
||||
- :cpp:func:`spi_flash_mmap` maps a region of physical flash addresses into instruction space or data space of the CPU
|
||||
- :cpp:func:`spi_flash_munmap` unmaps previously mapped region
|
||||
- :cpp:func:`esp_partition_mmap` maps part of a partition into the instruction space or data space of the CPU
|
||||
Memory mapping API are declared in ``esp_spi_flash.h`` and ``esp_partition.h``:
|
||||
|
||||
- :cpp:func:`spi_flash_mmap` maps a region of physical flash addresses into instruction space or data space of the CPU.
|
||||
- :cpp:func:`spi_flash_munmap` unmaps previously mapped region.
|
||||
- :cpp:func:`esp_partition_mmap` maps part of a partition into the instruction space or data space of the CPU.
|
||||
|
||||
Differences between :cpp:func:`spi_flash_mmap` and :cpp:func:`esp_partition_mmap` are as follows:
|
||||
|
||||
- :cpp:func:`spi_flash_mmap` must be given a 64KB aligned physical address
|
||||
- :cpp:func:`esp_partition_mmap` may be given any arbitrary offset within the partition,
|
||||
it will adjust returned pointer to mapped memory as necessary
|
||||
- :cpp:func:`spi_flash_mmap` must be given a 64KB aligned physical address.
|
||||
- :cpp:func:`esp_partition_mmap` may be given any arbitrary offset within the partition, it will adjust the returned pointer to mapped memory as necessary
|
||||
|
||||
Note that because memory mapping happens in 64KB blocks, it may be possible to
|
||||
read data outside of the partition provided to ``esp_partition_mmap``.
|
||||
Note that since memory mapping happens in 64KB blocks, it may be possible to read data outside of the partition provided to ``esp_partition_mmap``.
|
||||
|
||||
@@ -4,18 +4,17 @@ Virtual filesystem component
|
||||
Overview
|
||||
--------
|
||||
|
||||
Virtual filesystem (VFS) component provides a unified interface for drivers which can perform operations on file-like objects. This can be a real filesystems (FAT, SPIFFS, etc.), or device drivers which exposes file-like interface.
|
||||
Virtual filesystem (VFS) component provides a unified interface for drivers which can perform operations on file-like objects. These can be real filesystems (FAT, SPIFFS, etc.) or device drivers which provide a file-like interface.
|
||||
|
||||
This component allows C library functions, such as fopen and fprintf, to work with FS drivers. At high level, each FS driver is associated with some path prefix. When one of C library functions needs to open a file, VFS component searches for the FS driver associated with the file's path, and forwards the call to that driver. VFS also forwards read, write, and other calls for the given file to the same FS driver.
|
||||
This component allows C library functions, such as fopen and fprintf, to work with FS drivers. At a high level, each FS driver is associated with some path prefix. When one of C library functions needs to open a file, the VFS component searches for the FS driver associated with the file path and forwards the call to that driver. VFS also forwards read, write, and other calls for the given file to the same FS driver.
|
||||
|
||||
For example, one can register a FAT filesystem driver with the ``/fat`` prefix and call ``fopen("/fat/file.txt", "w")``. The VFS component will then call the function ``open`` of the FAT driver and pass the argument ``/file.txt`` to it together with appropriate mode flags. All subsequent calls to C library functions for the returned ``FILE*`` stream will also be forwarded to the FAT driver.
|
||||
|
||||
For example, one can register a FAT filesystem driver with ``/fat`` prefix, and call ``fopen("/fat/file.txt", "w")``. VFS component will then call ``open`` function of FAT driver and pass ``/file.txt`` argument to it (and appropriate mode flags). All subsequent calls to C library functions for the returned ``FILE*`` stream will also be forwarded to the FAT driver.
|
||||
|
||||
FS registration
|
||||
---------------
|
||||
|
||||
|
||||
|
||||
To register an FS driver, application needs to define in instance of :cpp:type:`esp_vfs_t` structure and populate it with function pointers to FS APIs:
|
||||
To register an FS driver, an application needs to define an instance of the :cpp:type:`esp_vfs_t` structure and populate it with function pointers to FS APIs:
|
||||
|
||||
.. highlight:: c
|
||||
|
||||
@@ -32,9 +31,9 @@ To register an FS driver, application needs to define in instance of :cpp:type:`
|
||||
|
||||
ESP_ERROR_CHECK(esp_vfs_register("/data", &myfs, NULL));
|
||||
|
||||
Depending on the way FS driver declares its APIs, either ``read``, ``write``, etc., or ``read_p``, ``write_p``, etc. should be used.
|
||||
Depending on the way how the FS driver declares its API functions, either ``read``, ``write``, etc., or ``read_p``, ``write_p``, etc., should be used.
|
||||
|
||||
Case 1: API functions are declared without an extra context pointer (FS driver is a singleton)::
|
||||
Case 1: API functions are declared without an extra context pointer (the FS driver is a singleton)::
|
||||
|
||||
ssize_t myfs_write(int fd, const void * data, size_t size);
|
||||
|
||||
@@ -46,7 +45,7 @@ Case 1: API functions are declared without an extra context pointer (FS driver i
|
||||
// When registering FS, context pointer (third argument) is NULL:
|
||||
ESP_ERROR_CHECK(esp_vfs_register("/data", &myfs, NULL));
|
||||
|
||||
Case 2: API functions are declared with an extra context pointer (FS driver supports multiple instances)::
|
||||
Case 2: API functions are declared with an extra context pointer (the FS driver supports multiple instances)::
|
||||
|
||||
ssize_t myfs_write(myfs_t* fs, int fd, const void * data, size_t size);
|
||||
|
||||
@@ -68,8 +67,8 @@ Synchronous input/output multiplexing
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If you want to use synchronous input/output multiplexing by :cpp:func:`select`
|
||||
then you need to register the VFS with :cpp:func:`start_select` and
|
||||
:cpp:func:`end_select` functions similarly to the following example:
|
||||
then you need to register VFS with the functions :cpp:func:`start_select` and
|
||||
:cpp:func:`end_select` similar to the following example:
|
||||
|
||||
.. highlight:: c
|
||||
|
||||
@@ -85,22 +84,27 @@ detection of read/write/error conditions on file descriptors belonging to the
|
||||
given VFS. :cpp:func:`end_select` is called to stop/deinitialize/free the
|
||||
environment which was setup by :cpp:func:`start_select`. Please refer to the
|
||||
reference implementation for the UART peripheral in
|
||||
:component_file:`vfs/vfs_uart.c` and most particularly to functions
|
||||
:cpp:func:`esp_vfs_dev_uart_register`, :cpp:func:`uart_start_select` and
|
||||
:component_file:`vfs/vfs_uart.c` and most particularly to the functions
|
||||
:cpp:func:`esp_vfs_dev_uart_register`, :cpp:func:`uart_start_select`, and
|
||||
:cpp:func:`uart_end_select`.
|
||||
|
||||
Examples demonstrating the use of :cpp:func:`select` with VFS file descriptors
|
||||
are the :example:`peripherals/uart_select` and the :example:`system/select`
|
||||
examples.
|
||||
Please check the following examples that demonstrate the use of :cpp:func:`select` with VFS file descriptors:
|
||||
- :example:`peripherals/uart_select`
|
||||
- :example:`system/select`
|
||||
|
||||
<<<<<<< HEAD
|
||||
If :cpp:func:`select` is used for socket file descriptors only then one can
|
||||
enable the :envvar:`CONFIG_LWIP_USE_ONLY_LWIP_SELECT` option which can reduce the code
|
||||
=======
|
||||
If you use :cpp:func:`select` for socket file descriptors, you can enable the :envvar:`CONFIG_LWIP_USE_ONLY_LWIP_SELECT` option to reduce the code
|
||||
>>>>>>> afc2fdf27... Review all the files in the esp-idf's api_ref/storage directory
|
||||
size and improve performance.
|
||||
|
||||
|
||||
Paths
|
||||
-----
|
||||
|
||||
Each registered FS has a path prefix associated with it. This prefix may be considered a "mount point" of this partition.
|
||||
Each registered FS has a path prefix associated with it. This prefix can be considered as a "mount point" of this partition.
|
||||
|
||||
In case when mount points are nested, the mount point with the longest matching path prefix is used when opening the file. For instance, suppose that the following filesystems are registered in VFS:
|
||||
|
||||
@@ -111,45 +115,49 @@ Then:
|
||||
|
||||
- FS 1 will be used when opening a file called ``/data/log.txt``
|
||||
- FS 2 will be used when opening a file called ``/data/static/index.html``
|
||||
- Even if ``/index.html"`` doesn't exist in FS 2, FS 1 will *not* be searched for ``/static/index.html``.
|
||||
- Even if ``/index.html"`` does not exist in FS 2, FS 1 will *not* be searched for ``/static/index.html``.
|
||||
|
||||
As a general rule, mount point names must start with the path separator (``/``) and must contain at least one character after path separator. However an empty mount point name is also supported, and may be used in cases when application needs to provide "fallback" filesystem, or override VFS functionality altogether. Such filesystem will be used if no prefix matches the path given.
|
||||
As a general rule, mount point names must start with the path separator (``/``) and must contain at least one character after path separator. However, an empty mount point name is also supported and might be used in cases when an application needs to provide a "fallback" filesystem or to override VFS functionality altogether. Such filesystem will be used if no prefix matches the path given.
|
||||
|
||||
VFS does not handle dots (``.``) in path names in any special way. VFS does not treat ``..`` as a reference to the parent directory. I.e. in the above example, using a path ``/data/static/../log.txt`` will not result in a call to FS 1 to open ``/log.txt``. Specific FS drivers (such as FATFS) may handle dots in file names differently.
|
||||
VFS does not handle dots (``.``) in path names in any special way. VFS does not treat ``..`` as a reference to the parent directory. In the above example, using a path ``/data/static/../log.txt`` will not result in a call to FS 1 to open ``/log.txt``. Specific FS drivers (such as FATFS) might handle dots in file names differently.
|
||||
|
||||
When opening files, FS driver will only be given relative path to files. For example:
|
||||
When opening files, the FS driver receives only relative paths to files. For example:
|
||||
|
||||
- ``myfs`` driver is registered with ``/data`` as path prefix
|
||||
- and application calls ``fopen("/data/config.json", ...)``
|
||||
- then VFS component will call ``myfs_open("/config.json", ...)``.
|
||||
- ``myfs`` driver will open ``/config.json`` file
|
||||
1. The ``myfs`` driver is registered with ``/data`` as a path prefix.
|
||||
2. The application calls ``fopen("/data/config.json", ...)``.
|
||||
3. The VFS component calls ``myfs_open("/config.json", ...)``.
|
||||
4. The ``myfs`` driver opens the ``/config.json`` file.
|
||||
|
||||
VFS does not impose any limit on total file path length, but it does limit the FS path prefix to ``ESP_VFS_PATH_MAX`` characters. Individual FS drivers may have their own filename length limitations.
|
||||
|
||||
VFS doesn't impose a limit on total file path length, but it does limit FS path prefix to ``ESP_VFS_PATH_MAX`` characters. Individual FS drivers may have their own filename length limitations.
|
||||
|
||||
File descriptors
|
||||
----------------
|
||||
|
||||
File descriptors are small positive integers from ``0`` to ``FD_SETSIZE - 1`` where ``FD_SETSIZE`` is defined in newlib's ``sys/types.h``. The largest file descriptors (configured by ``CONFIG_LWIP_MAX_SOCKETS``) are reserved for sockets. The VFS component contains a lookup-table called ``s_fd_table`` for mapping global file descriptors to VFS driver indexes registered in the ``s_vfs`` array.
|
||||
File descriptors are small positive integers from ``0`` to ``FD_SETSIZE - 1``, where ``FD_SETSIZE`` is defined in newlib's ``sys/types.h``. The largest file descriptors (configured by ``CONFIG_LWIP_MAX_SOCKETS``) are reserved for sockets. The VFS component contains a lookup-table called ``s_fd_table`` for mapping global file descriptors to VFS driver indexes registered in the ``s_vfs`` array.
|
||||
|
||||
|
||||
Standard IO streams (stdin, stdout, stderr)
|
||||
-------------------------------------------
|
||||
|
||||
If "UART for console output" menuconfig option is not set to "None", then ``stdin``, ``stdout``, and ``stderr`` are configured to read from, and write to, a UART. It is possible to use UART0 or UART1 for standard IO. By default, UART0 is used, with 115200 baud rate, TX pin is GPIO1 and RX pin is GPIO3. These parameters can be changed in menuconfig.
|
||||
If the menuconfig option ``UART for console output`` is not set to ``None``, then ``stdin``, ``stdout``, and ``stderr`` are configured to read from, and write to, a UART. It is possible to use UART0 or UART1 for standard IO. By default, UART0 is used with 115200 baud rate; TX pin is GPIO1; RX pin is GPIO3. These parameters can be changed in menuconfig.
|
||||
|
||||
Writing to ``stdout`` or ``stderr`` will send characters to the UART transmit FIFO. Reading from ``stdin`` will retrieve characters from the UART receive FIFO.
|
||||
|
||||
By default, VFS uses simple functions for reading from and writing to UART. Writes busy-wait until all data is put into UART FIFO, and reads are non-blocking, returning only the data present in the FIFO. Because of this non-blocking read behavior, higher level C library calls, such as ``fscanf("%d\n", &var);`` may not have desired results.
|
||||
By default, VFS uses simple functions for reading from and writing to UART. Writes busy-wait until all data is put into UART FIFO, and reads are non-blocking, returning only the data present in the FIFO. Due to this non-blocking read behavior, higher level C library calls, such as ``fscanf("%d\n", &var);``, might not have desired results.
|
||||
|
||||
Applications which use UART driver may instruct VFS to use the driver's interrupt driven, blocking read and write functions instead. This can be done using a call to ``esp_vfs_dev_uart_use_driver`` function. It is also possible to revert to the basic non-blocking functions using a call to ``esp_vfs_dev_uart_use_nonblocking``.
|
||||
Applications which use the UART driver can instruct VFS to use the driver's interrupt driven, blocking read and write functions instead. This can be done using a call to the ``esp_vfs_dev_uart_use_driver`` function. It is also possible to revert to the basic non-blocking functions using a call to ``esp_vfs_dev_uart_use_nonblocking``.
|
||||
|
||||
VFS also provides optional newline conversion feature for input and output. Internally, most applications send and receive lines terminated by LF (''\n'') character. Different terminal programs may require different line termination, such as CR or CRLF. Applications can configure this separately for input and output either via menuconfig, or by calls to ``esp_vfs_dev_uart_set_rx_line_endings`` and ``esp_vfs_dev_uart_set_tx_line_endings`` functions.
|
||||
VFS also provides an optional newline conversion feature for input and output. Internally, most applications send and receive lines terminated by the LF (''\n'') character. Different terminal programs may require different line termination, such as CR or CRLF. Applications can configure this separately for input and output either via menuconfig, or by calls to the functions ``esp_vfs_dev_uart_set_rx_line_endings`` and ``esp_vfs_dev_uart_set_tx_line_endings``.
|
||||
|
||||
|
||||
|
||||
Standard streams and FreeRTOS tasks
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
``FILE`` objects for ``stdin``, ``stdout``, and ``stderr`` are shared between all FreeRTOS tasks, but the pointers to these objects are are stored in per-task ``struct _reent``. The following code:
|
||||
``FILE`` objects for ``stdin``, ``stdout``, and ``stderr`` are shared between all FreeRTOS tasks, but the pointers to these objects are stored in per-task ``struct _reent``.
|
||||
|
||||
The following code is transferred to ``fprintf(__getreent()->_stderr, "42\n");`` by the preprocessor:
|
||||
|
||||
.. highlight:: c
|
||||
|
||||
@@ -157,14 +165,11 @@ Standard streams and FreeRTOS tasks
|
||||
|
||||
fprintf(stderr, "42\n");
|
||||
|
||||
actually is translated to to this (by the preprocessor)::
|
||||
|
||||
fprintf(__getreent()->_stderr, "42\n");
|
||||
|
||||
where the ``__getreent()`` function returns a per-task pointer to ``struct _reent`` (:component_file:`newlib/include/sys/reent.h#L370-L417`). This structure is allocated on the TCB of each task. When a task is initialized, ``_stdin``, ``_stdout`` and ``_stderr`` members of ``struct _reent`` are set to the values of ``_stdin``, ``_stdout`` and ``_stderr`` of ``_GLOBAL_REENT`` (i.e. the structure which is used before FreeRTOS is started).
|
||||
The ``__getreent()`` function returns a per-task pointer to ``struct _reent`` (:component_file:`newlib/include/sys/reent.h#L370-L417`). This structure is allocated on the TCB of each task. When a task is initialized, ``_stdin``, ``_stdout``, and ``_stderr`` members of ``struct _reent`` are set to the values of ``_stdin``, ``_stdout``, and ``_stderr`` of ``_GLOBAL_REENT`` (i.e., the structure which is used before FreeRTOS is started).
|
||||
|
||||
Such a design has the following consequences:
|
||||
|
||||
- It is possible to set ``stdin``, ``stdout``, and ``stderr`` for any given task without affecting other tasks, e.g. by doing ``stdin = fopen("/dev/uart/1", "r")``.
|
||||
- Closing default ``stdin``, ``stdout``, or ``stderr`` using ``fclose`` will close the ``FILE`` stream object — this will affect all other tasks.
|
||||
- It is possible to set ``stdin``, ``stdout``, and ``stderr`` for any given task without affecting other tasks, e.g., by doing ``stdin = fopen("/dev/uart/1", "r")``.
|
||||
- Closing default ``stdin``, ``stdout``, or ``stderr`` using ``fclose`` will close the ``FILE`` stream object, which will affect all other tasks.
|
||||
- To change the default ``stdin``, ``stdout``, ``stderr`` streams for new tasks, modify ``_GLOBAL_REENT->_stdin`` (``_stdout``, ``_stderr``) before creating the task.
|
||||
|
||||
@@ -1,55 +1,47 @@
|
||||
Wear Levelling APIs
|
||||
===================
|
||||
Wear Levelling API
|
||||
==================
|
||||
|
||||
Overview
|
||||
--------
|
||||
Most of the flash devices and specially SPI flash devices that are used in ESP32
|
||||
have sector based organization and have limited amount of erase/modification cycles
|
||||
per memory sector. To avoid situation when one sector reach the limit of erases when
|
||||
other sectors was used not often, we have made a component that avoid this situation.
|
||||
The wear levelling component share the amount of erases between all sectors in the
|
||||
memory without user interaction.
|
||||
The wear_levelling component contains APIs related to reading, writing, erasing,
|
||||
memory mapping data in the external SPI flash through the partition component. It
|
||||
also has higher-level APIs which work with FAT filesystem defined in
|
||||
the :doc:`FAT filesystem </api-reference/storage/fatfs>`.
|
||||
Most of flash memory and especially SPI flash that is used in ESP32 has a sector-based organization and also has a limited number of erase/modification cycles per memory sector. The wear levelling component helps to distribute wear and tear among sectors more evenly without requiring any attention from the user.
|
||||
|
||||
The wear levelling component, together with FAT FS component, works with FAT FS sector size 4096
|
||||
bytes which is standard size of the flash devices. In this mode the component has best performance,
|
||||
but needs additional memoty in the RAM. To save internal memory the component has two additional modes
|
||||
to work with sector size 512 bytes: Performance and Safety modes. In Performance mode by erase sector
|
||||
operation data will be stored to the RAM, sector will be erased and then data will be stored
|
||||
back to the flash. If by this operation power off situation will occur, the complete 4096 bytes
|
||||
will be lost. To prevent this the Safety mode was implemented. In safety mode the data will be first
|
||||
stored to the flash and after sector will be erased, will be stored back. If power off situation will
|
||||
occur, after power on, the data will be recovered.
|
||||
By default defined the sector size 512 bytes and Performance mode. To change these values please use
|
||||
the configuration menu.
|
||||
The wear levelling component provides API functions related to reading, writing, erasing, and memory mapping of data in external SPI flash through the partition component. The component also has higher-level API functions which work with the FAT filesystem defined in :doc:`FAT filesystem </api-reference/storage/fatfs>`.
|
||||
|
||||
The wear levelling component, together with the FAT FS component, uses FAT FS sectors of 4096 bytes, which is a standard size for flash memory. With this size, the component shows the best performance but needs additional memory in RAM.
|
||||
|
||||
To save internal memory, the component has two additional modes which both use sectors of 512 bytes:
|
||||
|
||||
- **Performance mode.** Erase sector operation data is stored in RAM, the sector is erased, and then data is copied back to flash memory. However, if a device is powered off for any reason, all 4096 bytes of data is lost.
|
||||
- **Safety mode.** The data is first saved to flash memory, and after the sector is erased, the data is saved back. If a device is powered off, the data can be recovered as soon as the device boots up.
|
||||
|
||||
The default settings are as follows:
|
||||
- Sector size is 512 bytes
|
||||
- Performance mode
|
||||
|
||||
You can change the settings through the configuration menu.
|
||||
|
||||
|
||||
The wear levelling component does not cache data in RAM. Write and erase functions
|
||||
modify flash directly, and flash contents is consistent when the function returns.
|
||||
The wear levelling component does not cache data in RAM. The write and erase functions modify flash directly, and flash contents are consistent when the function returns.
|
||||
|
||||
|
||||
Wear Levelling access APIs
|
||||
--------------------------
|
||||
Wear Levelling access API functions
|
||||
-----------------------------------
|
||||
|
||||
This is the set of APIs for working with data in flash:
|
||||
This is the set of API functions for working with data in flash:
|
||||
|
||||
- ``wl_mount`` mount wear levelling module for defined partition
|
||||
- ``wl_unmount`` used to unmount levelling module
|
||||
- ``wl_erase_range`` used to erase range of addresses in flash
|
||||
- ``wl_write`` used to write data to the partition
|
||||
- ``wl_read`` used to read data from the partition
|
||||
- ``wl_size`` return size of avalible memory in bytes
|
||||
- ``wl_sector_size`` returns size of one sector
|
||||
- ``wl_mount`` - initializes the wear levelling module and mounts the specified partition
|
||||
- ``wl_unmount`` - unmounts the partition and deinitializes the wear levelling module
|
||||
- ``wl_erase_range`` - erases a range of addresses in flash
|
||||
- ``wl_write`` - writes data to a partition
|
||||
- ``wl_read`` - reads data from a partition
|
||||
- ``wl_size`` - returns the size of available memory in bytes
|
||||
- ``wl_sector_size`` - returns the size of one sector
|
||||
|
||||
As a rule, try to avoid using raw wear levelling functions and use filesystem-specific functions instead.
|
||||
|
||||
Generally, try to avoid using the raw wear levelling functions in favor of
|
||||
filesystem-specific functions.
|
||||
|
||||
Memory Size
|
||||
-----------
|
||||
|
||||
The memory size calculated in the wear Levelling module based on parameters of
|
||||
partition. The module use few sectors of flash for internal data.
|
||||
The memory size is calculated in the wear levelling module based on partition parameters. The module uses some sectors of flash for internal data.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user