Update spiffs 0.3.7 to and mkspiffs 0.1.3

Author: me-no-dev

cores/esp8266/spiffs/LICENSE

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2013-2015 Peter Andersson (pelleplutt1976<at>gmail.com)
+Copyright (c) 2013-2017 Peter Andersson (pelleplutt1976<at>gmail.com)
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of
 this software and associated documentation files (the "Software"), to deal in

cores/esp8266/spiffs/README.md

@@ -1,7 +1,9 @@
 # SPIFFS (SPI Flash File System) 
-**V0.3.4**
+**V0.3.7**
 
-Copyright (c) 2013-2016 Peter Andersson (pelleplutt1976 at gmail.com)
+[![Build Status](https://travis-ci.org/pellepl/spiffs.svg?branch=master)](https://travis-ci.org/pellepl/spiffs)
+
+Copyright (c) 2013-2017 Peter Andersson (pelleplutt1976 at gmail.com)
 
 For legal stuff, see [LICENSE](https://github.com/pellepl/spiffs/blob/master/LICENSE). Basically, you may do whatever you want with the source. Use, modify, sell, print it out, roll it and smoke it - as long as I won't be held responsible.
 
@@ -21,34 +23,98 @@ Spiffs is designed with following characteristics in mind:
  - Wear leveling
 
 
+## BUILDING
+
+`mkdir build; make`
+
+Otherwise, configure the `builddir` variable towards the top of `makefile` as something opposed to the default `build`. Sanity check on the host via `make test` and refer to `.travis.yml` for the official in-depth testing procedure. See the wiki for [integrating](https://github.com/pellepl/spiffs/wiki/Integrate-spiffs) spiffs into projects and [spiffsimg](https://github.com/nodemcu/nodemcu-firmware/tree/master/tools/spiffsimg) from [nodemcu](https://github.com/nodemcu) is a good example on the subject.
+
+
 ## FEATURES
 
 What spiffs does:
  - Specifically designed for low ram usage
  - Uses statically sized ram buffers, independent of number of files
  - Posix-like api: open, close, read, write, seek, stat, etc
- - It can be run on any NOR flash, not only SPI flash - theoretically also on embedded flash of an microprocessor
- - Multiple spiffs configurations can be run on same target - and even on same SPI flash device
+ - It can run on any NOR flash, not only SPI flash - theoretically also on embedded flash of a microprocessor
+ - Multiple spiffs configurations can run on same target - and even on same SPI flash device
  - Implements static wear leveling 
  - Built in file system consistency checks
+ - Highly configurable
 
 What spiffs does not:
  - Presently, spiffs does not support directories. It produces a flat structure. Creating a file with path *tmp/myfile.txt* will create a file called *tmp/myfile.txt* instead of a *myfile.txt* under directory *tmp*. 
- - It is not a realtime stack. One write operation might take much longer than another.
- - Poor scalability. Spiffs is intended for small memory devices - the normal sizes for SPI flashes. Going beyond ~128MB is probably a bad idea. This is a side effect of the design goal to use as little ram as possible.
+ - It is not a realtime stack. One write operation might last much longer than another.
+ - Poor scalability. Spiffs is intended for small memory devices - the normal sizes for SPI flashes. Going beyond ~128Mbyte is probably a bad idea. This is a side effect of the design goal to use as little ram as possible.
  - Presently, it does not detect or handle bad blocks.
+ - One configuration, one binary. There's no generic spiffs binary that handles all types of configurations.
 
 
 ## MORE INFO 
 
-See the [wiki](https://github.com/pellepl/spiffs/wiki) for configuring, integrating and using spiffs.
+See the [wiki](https://github.com/pellepl/spiffs/wiki) for [configuring](https://github.com/pellepl/spiffs/wiki/Configure-spiffs), [integrating](https://github.com/pellepl/spiffs/wiki/Integrate-spiffs), [using](https://github.com/pellepl/spiffs/wiki/Using-spiffs), and [optimizing](https://github.com/pellepl/spiffs/wiki/Performance-and-Optimizing) spiffs.
 
 For design, see [docs/TECH_SPEC](https://github.com/pellepl/spiffs/blob/master/docs/TECH_SPEC).
 
 For a generic spi flash driver, see [this](https://github.com/pellepl/spiflash_driver).
 
 ## HISTORY
 
+### 0.3.7
+- fixed prevent seeking to negative offsets #158
+- fixed file descriptor offsets not updated for multiple fds on same file #157
+- fixed cache page not closed for removed files #156
+- fixed a lseek bug when seeking exactly to end of a fully indexed first level LUT #148
+- fixed wear leveling issue #145
+- fixed attempt to write out of bounds in flash #130, 
+- set file offset when seeking over end #121 (thanks @sensslen)
+- fixed seeking in virgin files #120 (thanks @sensslen)
+- Optional file metadata #128 (thanks @cesanta)
+- AFL testing framework #100 #143 (thanks @pjsg)
+- Testframe updates
+
+New API functions:
+- `SPIFFS_update_meta, SPIFFS_fupdate_meta` - updates metadata for a file
+
+New config defines:
+- `SPIFFS_OBJ_META_LEN` - enable possibility to add extra metadata to files
+
+### 0.3.6
+- Fix range bug in index memory mapping #98
+- Add index memory mapping #97
+- Optimize SPIFFS_read for large files #96
+- Add temporal cache for opening files #95
+- More robust gc #93 (thanks @dismirlian)
+- Fixed a double write of same data in certain cache situations
+- Fixed an open bug in READ_ONLY builds
+- File not visible in SPIFFS_readdir #90 (thanks @benpicco-tmp)
+- Cache load code cleanup #92 (thanks @niclash)
+- Fixed lock/unlock asymmetry #88 #87 (thanks @JackJefferson, @dpruessner)
+- Testframe updates
+
+New API functions:
+- `SPIFFS_ix_map` - map index meta data to memory for a file
+- `SPIFFS_ix_unmap` - unmaps index meta data for a file
+- `SPIFFS_ix_remap` - changes file offset for index metadata map
+- `SPIFFS_bytes_to_ix_map_entries` - utility, get length of needed vector for given amount of bytes
+- `SPIFFS_ix_map_entries_to_bytes` - utility, get number of bytes a vector can represent given length
+ 
+New config defines:
+- `SPIFFS_IX_MAP` - enable possibility to map index meta data to memory for reading faster 
+- `SPIFFS_TEMPORAL_FD_CACHE` - enable temporal cache for opening files faster
+- `SPIFFS_TEMPORAL_CACHE_HIT_SCORE` - for tuning the temporal cache
+
+### 0.3.5
+- Fixed a bug in fs check
+- API returns actual error codes #84) (thanks @Nails)
+- Fix compiler warnings for non-gcc #83 #81 (thanks @Nails)
+- Unable to recover from full fs #82 (thanks @rojer)
+- Define SPIFFS_O_* flags #80
+- Problem with long filenames #79 (thanks @psjg)
+- Duplicate file name bug fix #74 (thanks @igrr)
+- SPIFFS_eof and SPIFFS_tell return wrong value #72 (thanks @ArtemPisarenko)
+- Bunch of testframe updates #77 #78 #86 (thanks @dpreussner, @psjg a.o)  
+
 ### 0.3.4
 - Added user callback file func.
 - Fixed a stat bug with obj id.

cores/esp8266/spiffs/spiffs.h

@@ -54,6 +54,15 @@ extern "C" {
 #define SPIFFS_ERR_RO_ABORTED_OPERATION -10033
 #define SPIFFS_ERR_PROBE_TOO_FEW_BLOCKS -10034
 #define SPIFFS_ERR_PROBE_NOT_A_FS       -10035
+#define SPIFFS_ERR_NAME_TOO_LONG        -10036
+
+#define SPIFFS_ERR_IX_MAP_UNMAPPED      -10037
+#define SPIFFS_ERR_IX_MAP_MAPPED        -10038
+#define SPIFFS_ERR_IX_MAP_BAD_RANGE     -10039
+
+#define SPIFFS_ERR_SEEK_BOUNDS          -10040
+
+
 #define SPIFFS_ERR_INTERNAL             -10050
 
 #define SPIFFS_ERR_TEST                 -10100
@@ -104,7 +113,7 @@ typedef enum {
   SPIFFS_CHECK_FIX_LOOKUP,
   SPIFFS_CHECK_DELETE_ORPHANED_INDEX,
   SPIFFS_CHECK_DELETE_PAGE,
-  SPIFFS_CHECK_DELETE_BAD_FILE,
+  SPIFFS_CHECK_DELETE_BAD_FILE
 } spiffs_check_report;
 
 /* file system check callback function */
@@ -123,15 +132,15 @@ typedef enum {
   /* the file has been updated or moved to another page */
   SPIFFS_CB_UPDATED,
   /* the file has been deleted */
-  SPIFFS_CB_DELETED,
+  SPIFFS_CB_DELETED
 } spiffs_fileop_type;
 
 /* file system listener callback function */
 typedef void (*spiffs_file_callback)(struct spiffs_t *fs, spiffs_fileop_type op, spiffs_obj_id obj_id, spiffs_page_ix pix);
 
 #ifndef SPIFFS_DBG
 #define SPIFFS_DBG(...) \
-    print(__VA_ARGS__)
+    printf(__VA_ARGS__)
 #endif
 #ifndef SPIFFS_GC_DBG
 #define SPIFFS_GC_DBG(...) printf(__VA_ARGS__)
@@ -145,20 +154,28 @@ typedef void (*spiffs_file_callback)(struct spiffs_t *fs, spiffs_fileop_type op,
 
 /* Any write to the filehandle is appended to end of the file */
 #define SPIFFS_APPEND                   (1<<0)
+#define SPIFFS_O_APPEND                 SPIFFS_APPEND
 /* If the opened file exists, it will be truncated to zero length before opened */
 #define SPIFFS_TRUNC                    (1<<1)
+#define SPIFFS_O_TRUNC                  SPIFFS_TRUNC
 /* If the opened file does not exist, it will be created before opened */
 #define SPIFFS_CREAT                    (1<<2)
+#define SPIFFS_O_CREAT                  SPIFFS_CREAT
 /* The opened file may only be read */
 #define SPIFFS_RDONLY                   (1<<3)
-/* The opened file may only be writted */
+#define SPIFFS_O_RDONLY                 SPIFFS_RDONLY
+/* The opened file may only be written */
 #define SPIFFS_WRONLY                   (1<<4)
-/* The opened file may be both read and writted */
+#define SPIFFS_O_WRONLY                 SPIFFS_WRONLY
+/* The opened file may be both read and written */
 #define SPIFFS_RDWR                     (SPIFFS_RDONLY | SPIFFS_WRONLY)
-/* Any writes to the filehandle will never be cached */
+#define SPIFFS_O_RDWR                   SPIFFS_RDWR
+/* Any writes to the filehandle will never be cached but flushed directly */
 #define SPIFFS_DIRECT                   (1<<5)
-/* If SPIFFS_CREAT and SPIFFS_EXCL are set, SPIFFS_open() shall fail if the file exists */
+#define SPIFFS_O_DIRECT                 SPIFFS_DIRECT
+/* If SPIFFS_O_CREAT and SPIFFS_O_EXCL are set, SPIFFS_open() shall fail if the file exists */
 #define SPIFFS_EXCL                     (1<<6)
+#define SPIFFS_O_EXCL                   SPIFFS_EXCL
 
 #define SPIFFS_SEEK_SET                 (0)
 #define SPIFFS_SEEK_CUR                 (1)
@@ -283,6 +300,9 @@ typedef struct {
   spiffs_obj_type type;
   spiffs_page_ix pix;
   u8_t name[SPIFFS_OBJ_NAME_LEN];
+#if SPIFFS_OBJ_META_LEN
+  u8_t meta[SPIFFS_OBJ_META_LEN];
+#endif
 } spiffs_stat;
 
 struct spiffs_dirent {
@@ -291,6 +311,9 @@ struct spiffs_dirent {
   spiffs_obj_type type;
   u32_t size;
   spiffs_page_ix pix;
+#if SPIFFS_OBJ_META_LEN
+  u8_t meta[SPIFFS_OBJ_META_LEN];
+#endif
 };
 
 typedef struct {
@@ -299,6 +322,21 @@ typedef struct {
   int entry;
 } spiffs_DIR;
 
+#if SPIFFS_IX_MAP
+
+typedef struct {
+  // buffer with looked up data pixes
+  spiffs_page_ix *map_buf;
+  // precise file byte offset
+  u32_t offset;
+  // start data span index of lookup buffer
+  spiffs_span_ix start_spix;
+  // end data span index of lookup buffer
+  spiffs_span_ix end_spix;
+} spiffs_ix_map;
+
+#endif
+
 // functions
 
 #if SPIFFS_USE_MAGIC && SPIFFS_USE_MAGIC_LENGTH && SPIFFS_SINGLETON==0
@@ -375,8 +413,8 @@ s32_t SPIFFS_creat(spiffs *fs, const char *path, spiffs_mode mode);
  * @param fs            the file system struct
  * @param path          the path of the new file
  * @param flags         the flags for the open command, can be combinations of
- *                      SPIFFS_APPEND, SPIFFS_TRUNC, SPIFFS_CREAT, SPIFFS_RD_ONLY,
- *                      SPIFFS_WR_ONLY, SPIFFS_RDWR, SPIFFS_DIRECT
+ *                      SPIFFS_O_APPEND, SPIFFS_O_TRUNC, SPIFFS_O_CREAT, SPIFFS_O_RDONLY,
+ *                      SPIFFS_O_WRONLY, SPIFFS_O_RDWR, SPIFFS_O_DIRECT, SPIFFS_O_EXCL
  * @param mode          ignored, for posix compliance
  */
 spiffs_file SPIFFS_open(spiffs *fs, const char *path, spiffs_flags flags, spiffs_mode mode);
@@ -496,6 +534,24 @@ s32_t SPIFFS_close(spiffs *fs, spiffs_file fh);
  */
 s32_t SPIFFS_rename(spiffs *fs, const char *old, const char *newPath);
 
+#if SPIFFS_OBJ_META_LEN
+/**
+ * Updates file's metadata
+ * @param fs            the file system struct
+ * @param path          path to the file
+ * @param meta          new metadata. must be SPIFFS_OBJ_META_LEN bytes long.
+ */
+s32_t SPIFFS_update_meta(spiffs *fs, const char *name, const void *meta);
+
+/**
+ * Updates file's metadata
+ * @param fs            the file system struct
+ * @param fh            file handle of the file
+ * @param meta          new metadata. must be SPIFFS_OBJ_META_LEN bytes long.
+ */
+s32_t SPIFFS_fupdate_meta(spiffs *fs, spiffs_file fh, const void *meta);
+#endif
+
 /**
  * Returns last error of last file operation.
  * @param fs            the file system struct
@@ -648,6 +704,85 @@ s32_t SPIFFS_tell(spiffs *fs, spiffs_file fh);
  */
 s32_t SPIFFS_set_file_callback_func(spiffs *fs, spiffs_file_callback cb_func);
 
+#if SPIFFS_IX_MAP
+
+/**
+ * Maps the first level index lookup to a given memory map.
+ * This will make reading big files faster, as the memory map will be used for
+ * looking up data pages instead of searching for the indices on the physical
+ * medium. When mapping, all affected indicies are found and the information is
+ * copied to the array.
+ * Whole file or only parts of it may be mapped. The index map will cover file
+ * contents from argument offset until and including arguments (offset+len).
+ * It is valid to map a longer range than the current file size. The map will
+ * then be populated when the file grows.
+ * On garbage collections and file data page movements, the map array will be
+ * automatically updated. Do not tamper with the map array, as this contains
+ * the references to the data pages. Modifying it from outside will corrupt any
+ * future readings using this file descriptor.
+ * The map will no longer be used when the file descriptor closed or the file
+ * is unmapped.
+ * This can be useful to get faster and more deterministic timing when reading
+ * large files, or when seeking and reading a lot within a file.
+ * @param fs      the file system struct
+ * @param fh      the file handle of the file to map
+ * @param map     a spiffs_ix_map struct, describing the index map
+ * @param offset  absolute file offset where to start the index map
+ * @param len     length of the mapping in actual file bytes
+ * @param map_buf the array buffer for the look up data - number of required
+ *                elements in the array can be derived from function
+ *                SPIFFS_bytes_to_ix_map_entries given the length
+ */
+s32_t SPIFFS_ix_map(spiffs *fs, spiffs_file fh, spiffs_ix_map *map,
+    u32_t offset, u32_t len, spiffs_page_ix *map_buf);
+
+/**
+ * Unmaps the index lookup from this filehandle. All future readings will
+ * proceed as normal, requiring reading of the first level indices from
+ * physical media.
+ * The map and map buffer given in function SPIFFS_ix_map will no longer be
+ * referenced by spiffs.
+ * It is not strictly necessary to unmap a file before closing it, as closing
+ * a file will automatically unmap it.
+ * @param fs      the file system struct
+ * @param fh      the file handle of the file to unmap
+ */
+s32_t SPIFFS_ix_unmap(spiffs *fs, spiffs_file fh);
+
+/**
+ * Moves the offset for the index map given in function SPIFFS_ix_map. Parts or
+ * all of the map buffer will repopulated.
+ * @param fs      the file system struct
+ * @param fh      the mapped file handle of the file to remap
+ * @param offset  new absolute file offset where to start the index map
+ */
+s32_t SPIFFS_ix_remap(spiffs *fs, spiffs_file fh, u32_t offs);
+
+/**
+ * Utility function to get number of spiffs_page_ix entries a map buffer must
+ * contain on order to map given amount of file data in bytes.
+ * See function SPIFFS_ix_map and SPIFFS_ix_map_entries_to_bytes.
+ * @param fs      the file system struct
+ * @param bytes   number of file data bytes to map
+ * @return        needed number of elements in a spiffs_page_ix array needed to
+ *                map given amount of bytes in a file
+ */
+s32_t SPIFFS_bytes_to_ix_map_entries(spiffs *fs, u32_t bytes);
+
+/**
+ * Utility function to amount of file data bytes that can be mapped when
+ * mapping a file with buffer having given number of spiffs_page_ix entries.
+ * See function SPIFFS_ix_map and SPIFFS_bytes_to_ix_map_entries.
+ * @param fs      the file system struct
+ * @param map_page_ix_entries   number of entries in a spiffs_page_ix array
+ * @return        amount of file data in bytes that can be mapped given a map
+ *                buffer having given amount of spiffs_page_ix entries
+ */
+s32_t SPIFFS_ix_map_entries_to_bytes(spiffs *fs, u32_t map_page_ix_entries);
+
+#endif // SPIFFS_IX_MAP
+
+
 #if SPIFFS_TEST_VISUALISATION
 /**
  * Prints out a visualization of the filesystem.