corefiles/2121: gdb crashes sometimes on huge segment mappings

[lfk at kerbit dot net]

>Number:         2121
>Category:       corefiles
>Synopsis:       gdb crashes sometimes on huge segment mappings
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    unassigned
>State:          open
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Wed May 03 20:28:01 GMT 2006
>Closed-Date:
>Last-Modified:
>Originator:     Lynn Kerby
>Release:        GDB 6.4 base
>Organization:
>Environment:
RedHat ES3/AS4 ix86-32bit
>Description:
I'm trying to debug core files (and live processes) that have large address spaces (> 2 gig data) and have found that gdb is unable to deal with some of the resulting core files.  I had been using the 6.1post packages from RedHat which contain fixes for largefile support in the BFD library and other areas and recently decided to try upgrading to the 6.4 release (the number of patches in 6.1post is quite large) because I would occasionally come across a process/core that GDB couldn't handle.

There is a fundamental issue with the code that handles segment mappings in the inferior process or core file when those mappings are larger than SSIZE_MAX (2G - 1 on x86/32bit).  The fread/fwrite stdio calls simply return 0 if the size (after being munged into a signed type) is < 0.  This results in both unreadable core files and gdb crashes at times.
>How-To-Repeat:
Create a program that mallocs a 2.2GB chunk of space (note a large memory RedHat ES/AS config may be required), run it under gdb.
>Fix:
Both the BFD library (cache.c) and gdb gcore (gcore.c) code need to be modified to breakup I/O requests larger than SSIZE_MAX.  There were also some issues I encountered with argument types (effectively converting size_t to ssize_t then later to 64bit file_ptr types) that I felt should be fixed.  The supplied patch isn't quite as general as it should be (sorry, but I've got to get back to the problem that led me down this path).
>Release-Note:
>Audit-Trail:
>Unformatted:
----gnatsweb-attachment----
Content-Type: application/octet-stream; name="gdb-6.4-largesegcore.patch"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="gdb-6.4-largesegcore.patch"

LS0tIGdkYi02LjQvZ2RiL3RhcmdldC5oLm9yaWcJMjAwNS0wOS0wNCAwOToxODoyMC4wMDAwMDAw
MDAgLTA3MDAKKysrIGdkYi02LjQvZ2RiL3RhcmdldC5oCTIwMDYtMDUtMDEgMTY6NTk6NDUuMDAw
MDAwMDAwIC0wNzAwCkBAIC01MzcsNyArNTM3LDcgQEAgZXh0ZXJuIGludCBkb194ZmVyX21lbW9y
eSAoQ09SRV9BRERSIG1lbQogCiBleHRlcm4gaW50IHRhcmdldF9yZWFkX3N0cmluZyAoQ09SRV9B
RERSLCBjaGFyICoqLCBpbnQsIGludCAqKTsKIAotZXh0ZXJuIGludCB0YXJnZXRfcmVhZF9tZW1v
cnkgKENPUkVfQUREUiBtZW1hZGRyLCBnZGJfYnl0ZSAqbXlhZGRyLCBpbnQgbGVuKTsKK2V4dGVy
biBpbnQgdGFyZ2V0X3JlYWRfbWVtb3J5IChDT1JFX0FERFIgbWVtYWRkciwgZ2RiX2J5dGUgKm15
YWRkciwgTE9OR0VTVCBsZW4pOwogCiBleHRlcm4gaW50IHRhcmdldF93cml0ZV9tZW1vcnkgKENP
UkVfQUREUiBtZW1hZGRyLCBjb25zdCBnZGJfYnl0ZSAqbXlhZGRyLAogCQkJCWludCBsZW4pOwot
LS0gZ2RiLTYuNC9nZGIvdGFyZ2V0LmMub3JpZwkyMDA1LTA5LTA0IDA5OjE4OjIwLjAwMDAwMDAw
MCAtMDcwMAorKysgZ2RiLTYuNC9nZGIvdGFyZ2V0LmMJMjAwNi0wNS0wMSAxNjo1OToxMS4wMDAw
MDAwMDAgLTA3MDAKQEAgLTg1OCwxMCArODU4LDEyIEBAIHRhcmdldF94ZmVyX3BhcnRpYWwgKHN0
cnVjdCB0YXJnZXRfb3BzICoKIAkJICAgICBVTE9OR0VTVCBvZmZzZXQsIExPTkdFU1QgbGVuKQog
ewogICBMT05HRVNUIHJldHZhbDsKKyAgTE9OR0VTVCBtYXhsZW47CiAKICAgZ2RiX2Fzc2VydCAo
b3BzLT50b194ZmVyX3BhcnRpYWwgIT0gTlVMTCk7CisgIG1heGxlbiA9IG1pbihTU0laRV9NQVgg
JiB+KDB4ZmZmKSwgbGVuKTsJLyogc3RkaW8gY2FuJ3QgZG8gPj0gMkdiIHhmZXJzICovCiAgIHJl
dHZhbCA9IG9wcy0+dG9feGZlcl9wYXJ0aWFsIChvcHMsIG9iamVjdCwgYW5uZXgsIHJlYWRidWYs
IHdyaXRlYnVmLAotCQkJCSBvZmZzZXQsIGxlbik7CisJCQkJIG9mZnNldCwgbWF4bGVuKTsKICAg
aWYgKHRhcmdldGRlYnVnKQogICAgIHsKICAgICAgIGNvbnN0IHVuc2lnbmVkIGNoYXIgKm15YWRk
ciA9IE5VTEw7CkBAIC05OTQsNyArOTk2LDcgQEAgeGZlcl91c2luZ19zdHJhdHVtIChlbnVtIHRh
cmdldF9vYmplY3QgbwogICAgZGVhbCB3aXRoIHBhcnRpYWwgcmVhZHMgc2hvdWxkIGNhbGwgdGFy
Z2V0X3JlYWRfbWVtb3J5X3BhcnRpYWwuICovCiAKIGludAotdGFyZ2V0X3JlYWRfbWVtb3J5IChD
T1JFX0FERFIgbWVtYWRkciwgZ2RiX2J5dGUgKm15YWRkciwgaW50IGxlbikKK3RhcmdldF9yZWFk
X21lbW9yeSAoQ09SRV9BRERSIG1lbWFkZHIsIGdkYl9ieXRlICpteWFkZHIsIExPTkdFU1QgbGVu
KQogewogICBpZiAodGFyZ2V0X3hmZXJfcGFydGlhbF9wICgpKQogICAgIHJldHVybiB4ZmVyX3Vz
aW5nX3N0cmF0dW0gKFRBUkdFVF9PQkpFQ1RfTUVNT1JZLCBOVUxMLAotLS0gZ2RiLTYuNC9iZmQv
YmZkLWluLmgub3JpZwkyMDA1LTA5LTMwIDA4OjM2OjM1LjAwMDAwMDAwMCAtMDcwMAorKysgZ2Ri
LTYuNC9iZmQvYmZkLWluLmgJMjAwNi0wNS0wMSAxNzoyMToyMy4wMDAwMDAwMDAgLTA3MDAKQEAg
LTY4OSw3ICs2ODksNyBAQCBleHRlcm4gaW50IGJmZF9nZXRfZWxmX3BoZHJzCiAgICB0aGUgcmVt
b3RlIG1lbW9yeS4gICovCiBleHRlcm4gYmZkICpiZmRfZWxmX2JmZF9mcm9tX3JlbW90ZV9tZW1v
cnkKICAgKGJmZCAqdGVtcGwsIGJmZF92bWEgZWhkcl92bWEsIGJmZF92bWEgKmxvYWRiYXNlcCwK
LSAgIGludCAoKnRhcmdldF9yZWFkX21lbW9yeSkgKGJmZF92bWEgdm1hLCBiZmRfYnl0ZSAqbXlh
ZGRyLCBpbnQgbGVuKSk7CisgICBpbnQgKCp0YXJnZXRfcmVhZF9tZW1vcnkpIChiZmRfdm1hIHZt
YSwgYmZkX2J5dGUgKm15YWRkciwgYmZkX3NpZ25lZF92bWEgbGVuKSk7CiAKIC8qIFJldHVybiB0
aGUgYXJjaF9zaXplIGZpZWxkIG9mIGFuIGVsZiBiZmQsIG9yIC0xIGlmIG5vdCBlbGYuICAqLwog
ZXh0ZXJuIGludCBiZmRfZ2V0X2FyY2hfc2l6ZQotLS0gZ2RiLTYuNC9iZmQvYmZkLWluMi5oLm9y
aWcJMjAwNS0xMC0yNSAxMDo0MDowOS4wMDAwMDAwMDAgLTA3MDAKKysrIGdkYi02LjQvYmZkL2Jm
ZC1pbjIuaAkyMDA2LTA1LTAxIDE3OjIyOjA1LjAwMDAwMDAwMCAtMDcwMApAQCAtNjk2LDcgKzY5
Niw3IEBAIGV4dGVybiBpbnQgYmZkX2dldF9lbGZfcGhkcnMKICAgIHRoZSByZW1vdGUgbWVtb3J5
LiAgKi8KIGV4dGVybiBiZmQgKmJmZF9lbGZfYmZkX2Zyb21fcmVtb3RlX21lbW9yeQogICAoYmZk
ICp0ZW1wbCwgYmZkX3ZtYSBlaGRyX3ZtYSwgYmZkX3ZtYSAqbG9hZGJhc2VwLAotICAgaW50ICgq
dGFyZ2V0X3JlYWRfbWVtb3J5KSAoYmZkX3ZtYSB2bWEsIGJmZF9ieXRlICpteWFkZHIsIGludCBs
ZW4pKTsKKyAgIGludCAoKnRhcmdldF9yZWFkX21lbW9yeSkgKGJmZF92bWEgdm1hLCBiZmRfYnl0
ZSAqbXlhZGRyLCBiZmRfc2lnbmVkX3ZtYSBsZW4pKTsKIAogLyogUmV0dXJuIHRoZSBhcmNoX3Np
emUgZmllbGQgb2YgYW4gZWxmIGJmZCwgb3IgLTEgaWYgbm90IGVsZi4gICovCiBleHRlcm4gaW50
IGJmZF9nZXRfYXJjaF9zaXplCi0tLSBnZGItNi40L2JmZC9jYWNoZS5jLm9yaWcJMjAwNi0wNS0w
MiAxOToxNjozNi4wMDAwMDAwMDAgLTA3MDAKKysrIGdkYi02LjQvYmZkL2NhY2hlLmMJMjAwNi0w
NS0wMiAxOTo0MTo1Ni4wMDAwMDAwMDAgLTA3MDAKQEAgLTQ0LDYgKzQ0LDE1IEBAIFNVQlNFQ1RJ
T04KICNpbmNsdWRlICJzeXNkZXAuaCIKICNpbmNsdWRlICJsaWJiZmQuaCIKICNpbmNsdWRlICJs
aWJpYmVydHkuaCIKKyNpbmNsdWRlIDxsaW1pdHMuaD4KKworI2lmbmRlZiBtaW4KKyNkZWZpbmUg
bWluKGEsIGIpICgoYSkgPCAoYikgPyAoYSkgOiAoYikpCisjZW5kaWYKKyNpZm5kZWYgbWF4Cisj
ZGVmaW5lIG1heChhLCBiKSAoKGEpID4gKGIpID8gKGEpIDogKGIpKQorI2VuZGlmCisKIAogLyog
SW4gc29tZSBjYXNlcyB3ZSBjYW4gb3B0aW1pemUgY2FjaGUgb3BlcmF0aW9uIHdoZW4gcmVvcGVu
aW5nIGZpbGVzLgogICAgRm9yIGluc3RhbmNlLCBhIGZsdXNoIGlzIGVudGlyZWx5IHVubmVjZXNz
YXJ5IGlmIHRoZSBmaWxlIGlzIGFscmVhZHkKQEAgLTI2Nyw4ICsyNzYsOSBAQCBjYWNoZV9ic2Vl
ayAoc3RydWN0IGJmZCAqYWJmZCwgZmlsZV9wdHIgCiBzdGF0aWMgZmlsZV9wdHIKIGNhY2hlX2Jy
ZWFkIChzdHJ1Y3QgYmZkICphYmZkLCB2b2lkICpidWYsIGZpbGVfcHRyIG5ieXRlcykKIHsKKyAg
ZmlsZV9wdHIgbnJlYWQgPSAwOworICBmaWxlX3B0ciByYnl0ZXM7CiAgIEZJTEUgKmY7Ci0gIGZp
bGVfcHRyIG5yZWFkOwogICAvKiBGSVhNRSAtIHRoaXMgbG9va3MgbGlrZSBhbiBvcHRpbWl6YXRp
b24sIGJ1dCBpdCdzIHJlYWxseSB0byBjb3ZlcgogICAgICB1cCBmb3IgYSBmZWF0dXJlIG9mIHNv
bWUgT1NzIChub3Qgc29sYXJpcyAtIHNpZ2gpIHRoYXQKICAgICAgbGQvcGUtZGxsLmMgdGFrZXMg
YWR2YW50YWdlIG9mIChhcHBhcmVudGx5KSB3aGVuIGl0IGNyZWF0ZXMgQkZEcwpAQCAtMjk4LDcg
KzMwOCwxMyBAQCBjYWNoZV9icmVhZCAoc3RydWN0IGJmZCAqYWJmZCwgdm9pZCAqYnVmCiAgICAg
ICByZXR1cm4gLTE7CiAgICAgfQogI2Vsc2UKLSAgbnJlYWQgPSBmcmVhZCAoYnVmLCAxLCBuYnl0
ZXMsIGYpOworICAvKiBoYW5kbGUgdG9vIGxhcmdlIHJlYWQvd3JpdGUgY2FzZSAqLworICByYnl0
ZXMgPSBtaW4oU1NJWkVfTUFYICYgfjB4ZmZmLCBuYnl0ZXMpOworICBpZiAobmJ5dGVzID4gcmJ5
dGVzKSB7CisgICAgbnJlYWQgPSBmcmVhZCAoYnVmLCAxLCBTU0laRV9NQVggJiB+MHhmZmYsIGYp
OworICAgIHJieXRlcyA9IG5ieXRlcyAtIG5yZWFkOworICB9CisgIG5yZWFkICs9IGZyZWFkIChi
dWYsIDEsIHJieXRlcywgZik7CiAgIC8qIFNldCBiZmRfZXJyb3IgaWYgd2UgZGlkIG5vdCByZWFk
IGFzIG11Y2ggZGF0YSBhcyB3ZSBleHBlY3RlZC4gIElmCiAgICAgIHRoZSByZWFkIGZhaWxlZCBk
dWUgdG8gYW4gZXJyb3Igc2V0IHRoZSBiZmRfZXJyb3Jfc3lzdGVtX2NhbGwsCiAgICAgIGVsc2Ug
c2V0IGJmZF9lcnJvcl9maWxlX3RydW5jYXRlZC4gICovCkBAIC0zMTQsMTEgKzMzMCwyMCBAQCBj
YWNoZV9icmVhZCAoc3RydWN0IGJmZCAqYWJmZCwgdm9pZCAqYnVmCiBzdGF0aWMgZmlsZV9wdHIK
IGNhY2hlX2J3cml0ZSAoc3RydWN0IGJmZCAqYWJmZCwgY29uc3Qgdm9pZCAqd2hlcmUsIGZpbGVf
cHRyIG5ieXRlcykKIHsKLSAgZmlsZV9wdHIgbndyaXRlOworICBmaWxlX3B0ciBud3JpdGUgPSAw
OworICBmaWxlX3B0ciB3Ynl0ZXM7CiAgIEZJTEUgKmYgPSBiZmRfY2FjaGVfbG9va3VwIChhYmZk
LCAwKTsKICAgaWYgKGYgPT0gTlVMTCkKICAgICByZXR1cm4gMDsKLSAgbndyaXRlID0gZndyaXRl
ICh3aGVyZSwgMSwgbmJ5dGVzLCBmKTsKKworICAvKiBoYW5kbGUgdG9vIGxhcmdlIHJlYWQvd3Jp
dGUgY2FzZSAqLworICB3Ynl0ZXMgPSBtaW4oU1NJWkVfTUFYICYgfjB4ZmZmLCBuYnl0ZXMpOwor
ICBpZiAobmJ5dGVzID4gd2J5dGVzKSB7CisgICAgbndyaXRlID0gZndyaXRlICh3aGVyZSwgMSwg
U1NJWkVfTUFYICYgfjB4ZmZmLCBmKTsKKyAgICB3Ynl0ZXMgPSBuYnl0ZXMgLSBud3JpdGU7Cisg
IH0KKworICBud3JpdGUgKz0gZndyaXRlICh3aGVyZSwgMSwgd2J5dGVzLCBmKTsKICAgaWYgKG53
cml0ZSA8IG5ieXRlcyAmJiBmZXJyb3IgKGYpKQogICAgIHsKICAgICAgIGJmZF9zZXRfZXJyb3Ig
KGJmZF9lcnJvcl9zeXN0ZW1fY2FsbCk7Ci0tLSBnZGItNi40L2JmZC9lbGYuYy5vcmlnCTIwMDYt
MDUtMDEgMTc6MzA6MjkuMDAwMDAwMDAwIC0wNzAwCisrKyBnZGItNi40L2JmZC9lbGYuYwkyMDA2
LTA1LTAxIDE3OjMxOjAwLjAwMDAwMDAwMCAtMDcwMApAQCAtODE2NSw3ICs4MTY1LDcgQEAgYmZk
X2VsZl9iZmRfZnJvbV9yZW1vdGVfbWVtb3J5CiAgIChiZmQgKnRlbXBsLAogICAgYmZkX3ZtYSBl
aGRyX3ZtYSwKICAgIGJmZF92bWEgKmxvYWRiYXNlcCwKLSAgIGludCAoKnRhcmdldF9yZWFkX21l
bW9yeSkgKGJmZF92bWEsIGJmZF9ieXRlICosIGludCkpCisgICBpbnQgKCp0YXJnZXRfcmVhZF9t
ZW1vcnkpIChiZmRfdm1hLCBiZmRfYnl0ZSAqLCBiZmRfc2lnbmVkX3ZtYSkpCiB7CiAgIHJldHVy
biAoKmdldF9lbGZfYmFja2VuZF9kYXRhICh0ZW1wbCktPmVsZl9iYWNrZW5kX2JmZF9mcm9tX3Jl
bW90ZV9tZW1vcnkpCiAgICAgKHRlbXBsLCBlaGRyX3ZtYSwgbG9hZGJhc2VwLCB0YXJnZXRfcmVh
ZF9tZW1vcnkpOwotLS0gZ2RiLTYuNC9iZmQvZWxmLWJmZC5oLm9yaWcJMjAwNS0xMC0yNSAwOTox
OTowNi4wMDAwMDAwMDAgLTA3MDAKKysrIGdkYi02LjQvYmZkL2VsZi1iZmQuaAkyMDA2LTA1LTAx
IDE3OjUwOjMyLjAwMDAwMDAwMCAtMDcwMApAQCAtOTc0LDcgKzk3NCw3IEBAIHN0cnVjdCBlbGZf
YmFja2VuZF9kYXRhCiAgICAgIHNlZSBlbGYuYywgZWxmY29kZS5oLiAgKi8KICAgYmZkICooKmVs
Zl9iYWNrZW5kX2JmZF9mcm9tX3JlbW90ZV9tZW1vcnkpCiAgICAgIChiZmQgKnRlbXBsLCBiZmRf
dm1hIGVoZHJfdm1hLCBiZmRfdm1hICpsb2FkYmFzZXAsCi0gICAgICBpbnQgKCp0YXJnZXRfcmVh
ZF9tZW1vcnkpIChiZmRfdm1hIHZtYSwgYmZkX2J5dGUgKm15YWRkciwgaW50IGxlbikpOworICAg
ICAgaW50ICgqdGFyZ2V0X3JlYWRfbWVtb3J5KSAoYmZkX3ZtYSB2bWEsIGJmZF9ieXRlICpteWFk
ZHIsIGJmZF9zaWduZWRfdm1hIGxlbikpOwogCiAgIC8qIFRoaXMgZnVuY3Rpb24gaXMgdXNlZCBi
eSBgX2JmZF9lbGZfZ2V0X3N5bnRoZXRpY19zeW10YWInOwogICAgICBzZWUgZWxmLmMuICAqLwpA
QCAtMTg1MCwxMCArMTg1MCwxMCBAQCBleHRlcm4gY2hhciAqZWxmY29yZV93cml0ZV9sd3BzdGF0
dXMKIAogZXh0ZXJuIGJmZCAqX2JmZF9lbGYzMl9iZmRfZnJvbV9yZW1vdGVfbWVtb3J5CiAgIChi
ZmQgKnRlbXBsLCBiZmRfdm1hIGVoZHJfdm1hLCBiZmRfdm1hICpsb2FkYmFzZXAsCi0gICBpbnQg
KCp0YXJnZXRfcmVhZF9tZW1vcnkpIChiZmRfdm1hLCBiZmRfYnl0ZSAqLCBpbnQpKTsKKyAgIGlu
dCAoKnRhcmdldF9yZWFkX21lbW9yeSkgKGJmZF92bWEsIGJmZF9ieXRlICosIGJmZF9zaWduZWRf
dm1hKSk7CiBleHRlcm4gYmZkICpfYmZkX2VsZjY0X2JmZF9mcm9tX3JlbW90ZV9tZW1vcnkKICAg
KGJmZCAqdGVtcGwsIGJmZF92bWEgZWhkcl92bWEsIGJmZF92bWEgKmxvYWRiYXNlcCwKLSAgIGlu
dCAoKnRhcmdldF9yZWFkX21lbW9yeSkgKGJmZF92bWEsIGJmZF9ieXRlICosIGludCkpOworICAg
aW50ICgqdGFyZ2V0X3JlYWRfbWVtb3J5KSAoYmZkX3ZtYSwgYmZkX2J5dGUgKiwgYmZkX3NpZ25l
ZF92bWEpKTsKIAogLyogTGFyZ2UgY29tbW9uIHNlY3Rpb24uICAqLwogZXh0ZXJuIGFzZWN0aW9u
IF9iZmRfZWxmX2xhcmdlX2NvbV9zZWN0aW9uOwotLS0gZ2RiLTYuNC9iZmQvZWxmY29kZS5oLm9y
aWcJMjAwNS0wOS0yOCAwNzo1MzoyNC4wMDAwMDAwMDAgLTA3MDAKKysrIGdkYi02LjQvYmZkL2Vs
ZmNvZGUuaAkyMDA2LTA1LTAxIDE3OjU4OjM1LjAwMDAwMDAwMCAtMDcwMApAQCAtMTU3MCw3ICsx
NTcwLDcgQEAgTkFNRShfYmZkX2VsZixiZmRfZnJvbV9yZW1vdGVfbWVtb3J5KQogICAoYmZkICp0
ZW1wbCwKICAgIGJmZF92bWEgZWhkcl92bWEsCiAgICBiZmRfdm1hICpsb2FkYmFzZXAsCi0gICBp
bnQgKCp0YXJnZXRfcmVhZF9tZW1vcnkpIChiZmRfdm1hLCBiZmRfYnl0ZSAqLCBpbnQpKQorICAg
aW50ICgqdGFyZ2V0X3JlYWRfbWVtb3J5KSAoYmZkX3ZtYSwgYmZkX2J5dGUgKiwgYmZkX3NpZ25l
ZF92bWEpKQogewogICBFbGZfRXh0ZXJuYWxfRWhkciB4X2VoZHI7CS8qIEVsZiBmaWxlIGhlYWRl
ciwgZXh0ZXJuYWwgZm9ybSAqLwogICBFbGZfSW50ZXJuYWxfRWhkciBpX2VoZHI7CS8qIEVsZiBm
aWxlIGhlYWRlciwgaW50ZXJuYWwgZm9ybSAqLwo=