Re: [ECOS] JFFS2 "eating memory" when creating files

[Thomas Koeller <thomas dot koeller at baslerweb dot com>]

Hi Øyvind,

as I said, it depends on the size of your flash, the average size of your
files, the page size set in the linux compatibility package, and how you
use of your file system in general. Read David's paper on the design of
JFFS2.

The flash is managed as a sequential log of nodes that do not have a fixed
size. Every such node is represented in RAM by a struct of size 16 bytes
(24 bytes if malloc'ed because of malloc() granularity) of type struct
raw_node_ref. Their life times are tied to the life time of the node they
represent, which is from creation until being garbage-collected after
having been obsoleted.

Every change to the file system (even deletion of files) will be carried out
by writing new nodes to the log and will therefore lead to an increase in
RAM usage. Only garbage collection reduces it. Garbage collection is
triggered by the amount of unused flash dropping below five erase blocks.
If the file system is much larger than needed, the free space will be filled
with obsoleted nodes over time, which still consume RAM. It is therefore
unwise to create a file system much larger than the anticipated storage
requirements, even if flash space is not a concern.

The minimum node size allowed by JFFS2 is 128 Bytes. Divide the usable size
of your flash (total size minus the five erase blocks that JFFS2 reserves
for its own use) by this number and you will get a worst-case estimate of
the total number of nodes that will ever be created.

Of course you will only create that many nodes if you write your files in
tiny chunks. You can also estimate the minimum number of nodes needed to
entirely fill your file system with data. There are two limits on the size
of a node: the page size (set in the linux compatibility package) and half
the erase block size of your flash, whichever is smaller. You can use this
maximum node size to compute the minimum number of nodes that will be created.

Somewhere between these two extremes lies the number you are asking for. You can
reduce the figure by increasing the page size. But do not set it higher than
half the erase block size. Also, JFFS2 creates two static buffers of size
PAGE_SIZE each. Avoid writing files in small data chunks, if you can; use a 
sufficiently large buffer.

Using the pooled allocation method for the raw_node_ref structs will save
8 bytes per node, which can be substantial. It also avoids heap fragmentation,
and is more efficient in general. A sound strategy would be to estimate how
much RAM your program uses for other purposes, and dedicate the rest to
the pool. And don't worry: if you set it too small, the error you get will
be exactly the same as if malloc() runs out of heap space during a non-pooled
allocation.

The discussion above neglects a few facts, namely:
- there is some storage overhead introduced by node headers
- not all nodes are data nodes, some represent inodes, dirents, and so on
- the file system allocates some more data structure in RAM

David, since you are the main JFFS2 authority, I cc'ed this to you. Would you
mind confirming that I am not telling utter nonsense here, and correct me if I
got something wrong?

tk


Øyvind Harboe wrote:
> > As I already pointed out earlier, every operation that modifies the
> > contents of a JFFS2 file system writes new nodes to the log, which are
> > represented in RAM as a struct raw_node_ref. Hence the increase in RAM
> > usage is normal and expected behavior. RAM usage will continously
> > increase until garbage collection is triggered.
>
> I see.
>
> > Try using the pool option and watch the increase disappear.
>
> I guess I worry a bit about what size to set it to.
>
> What will happen if I set it too low?
>
> What is the right size for my application?
>
> "Read the source Luke" :-)
>
>
>
> Øyvind

-- 
--------------------------------------------------

Thomas Koeller, Software Development

Basler Vision Technologies
An der Strusbek 60-62
22926 Ahrensburg
Germany

Tel +49 (4102) 463-390
Fax +49 (4102) 463-46390

mailto:thomas.koeller@baslerweb.com
http://www.baslerweb.com

==============================


--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss