This is the mail archive of the libc-help@sourceware.org mailing list for the glibc project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Memory consumption of iconv


On Fri, Mar 21, 2014 at 7:53 AM, Oliver Becker <der.ole.becker@gmail.com> wrote:
> Hi,
>
> I am developing a server software which spawns up to thousands of client
> processes which use iconv_open (through Qt) to later transform strings between
> different encodings. This works like a charm of course but after a while I
> recognized that every process used about 1MB (non shared) memory for the iconv
> data.
> This allone is not that much but If you multiply it by the number of child
> processes, I get a huge size of multiple GB. If you take into account that I
> use the same locales in every child process this is not easy to understand.
>
> I searched through the iconv/gconv sources but didn't get any farther when
> coming to the point where external module functions get called which allocate
> the actual data.
>
> So my question is: Do the gconv_* functions which load the actual "big" (some
> KB) amount of data for transforming use shared memory segments (like shmget)
> and my system just doesn't report it? Or can't they use them because of some
> reason?
>
> My thought of using shared memory is to just decrease the overall memory
> footprint of my application from ~1-10GB to several MB.

You need to determine exactly where this 1MB is coming from.

The iconv converters are dynamically shared objects which are mmap'd into
the processes address space by the dynamic loader. The read-only data
and code for these DSOs only take up space once per loaded converter,
since the kernel shares the mapping across all other processes.

The problem might be that there is read-write data used for the specific
converter implementation and that might take up space. However, without
optimizing the converter code for size you won't make much gains here.

Suggested next steps:
* Determine exactly where your 1MB of non-shared memory came from.
* Try to determine if you can reduce that 1MB.

The use case you're presenting is not something that anyone has considered.

Cheers,
Carlos.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]