--- case c --------------------- plana85 file existed, so these same writes happened prior to the crash and were replayed by the ceph journal. 2013-06-07 18:13:34.297208 7f44210f7780 0 ceph version 0.61.2-43-g479350a (479350ae7df42cdca8584732d4e3e0b4341d1a1e), process ceph-osd, pid 1848 2013-06-07 18:18:07.875769 7f4417149700 15 filestore(/var/lib/ceph/osd/ceph-0) write 3.0_head/834c52b9/plana8021941-457/head//3 0~59 2013-06-07 18:18:07.875812 7f4417149700 10 filestore(/var/lib/ceph/osd/ceph-0) file did exist 2013-06-07 18:18:07.875858 7f4417149700 10 filestore(/var/lib/ceph/osd/ceph-0) before: size 2482415 blocks 2336 2013-06-07 18:18:07.875872 7f4417149700 20 filestore(/var/lib/ceph/osd/ceph-0) write lseek fd 24 2013-06-07 18:18:07.875896 7f4417149700 10 filestore(/var/lib/ceph/osd/ceph-0) after: size 2482415 blocks 2336 2013-06-07 18:18:07.875905 7f4417149700 10 filestore(/var/lib/ceph/osd/ceph-0) write 3.0_head/834c52b9/plana8021941-457/head//3 0~59 = 59 2013-06-07 18:18:07.876481 7f4417149700 15 filestore(/var/lib/ceph/osd/ceph-0) write 3.0_head/834c52b9/plana8021941-457/head//3 700466~445465 2013-06-07 18:18:07.876526 7f4417149700 10 filestore(/var/lib/ceph/osd/ceph-0) file did exist 2013-06-07 18:18:07.876574 7f4417149700 10 filestore(/var/lib/ceph/osd/ceph-0) before: size 2482415 blocks 2336 2013-06-07 18:18:07.876578 7f4417149700 20 filestore(/var/lib/ceph/osd/ceph-0) write lseek fd 24 2013-06-07 18:18:07.876963 7f4417149700 10 filestore(/var/lib/ceph/osd/ceph-0) after: size 2482415 blocks 2336 2013-06-07 18:18:07.876983 7f4417149700 10 filestore(/var/lib/ceph/osd/ceph-0) write 3.0_head/834c52b9/plana8021941-457/head//3 700466~445465 = 445465 2013-06-07 18:18:07.877223 7f4416147700 10 filestore(/var/lib/ceph/osd/ceph-0) before close: size 2482415 blocks 2336 2013-06-07 18:18:07.878159 7f4416948700 15 filestore(/var/lib/ceph/osd/ceph-0) write 3.0_head/834c52b9/plana8021941-457/head//3 1670161~581044 2013-06-07 18:18:07.878208 7f4416948700 10 filestore(/var/lib/ceph/osd/ceph-0) file did exist 2013-06-07 18:18:07.878258 7f4416948700 10 filestore(/var/lib/ceph/osd/ceph-0) before: size 2482415 blocks 2336 2013-06-07 18:18:07.878262 7f4416948700 20 filestore(/var/lib/ceph/osd/ceph-0) write lseek fd 25 2013-06-07 18:18:07.878776 7f4416948700 10 filestore(/var/lib/ceph/osd/ceph-0) after: size 2482415 blocks 2688 2013-06-07 18:18:07.878796 7f4416948700 10 filestore(/var/lib/ceph/osd/ceph-0) write 3.0_head/834c52b9/plana8021941-457/head//3 1670161~581044 = 581044 2013-06-07 18:18:07.879197 7f4416147700 10 filestore(/var/lib/ceph/osd/ceph-0) before close: size 2482415 blocks 2688 /var/lib/ceph/osd/ceph-0/current/3.0_head/DIR_9/plana8021941-457__head_834C52B9__3: 0: [0..7]: 244206232..244206239 1: [8..1351]: hole 2: [1352..2431]: 244252824..244253903 3: [2432..3255]: hole 4: [3256..4855]: 244254728..244256327 bad data starts at offset 1179648: ... 0011f820 f7 74 55 62 54 64 d4 60 8f a0 69 50 3e fd 4d 44 |.tUbTd.`..iP>.MD| 0011f830 35 d9 1e 8b 85 ef c2 e1 42 ed 0c 6a 07 c7 1f 5e |5.......B..j...^| 0011f840 3a c9 95 83 0e 6d 5f 5f 4c f1 6a 29 d2 a6 05 83 |:....m__L.j)....| 0011f850 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00120000 85 35 3b ca e0 ff 69 20 cf b5 8c c3 36 fe 40 82 |.5;...i ....6.@.| 00120010 ae 98 38 8b 37 9b 4f 92 ae 80 5f b4 ff e0 d4 18 |..8.7.O..._.....| 00120020 1a fd ff 59 a6 01 2a a6 76 07 6f a5 bf dd 48 1b |...Y..*.v.o...H.| 00120030 06 86 e5 4c da 80 40 23 46 ca f1 10 03 a4 c1 b4 |...L..@#F.......| ... 0012ffe0 ed de 09 aa 9d 76 97 ca 1f 94 65 02 09 ba 31 2b |.....v....e...1+| 0012fff0 07 5e 31 5d 8a 34 21 16 42 2c 1e 1c 4c 3c 34 22 |.^1].4!.B,..L<4"| 00130000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00197c10 00 ca 5e 4d aa d6 46 cc 8b 9c 71 a5 3b d0 b9 4f |..^M..F...q.;..O| 00197c20 94 8f 0b 64 02 00 56 25 d6 be 43 be 5a f8 a5 4a |...d..V%..C.Z..J| 00197c30 a8 54 af 94 3a 97 e7 a2 40 05 dc f2 b4 89 98 81 |.T..:...@.......| this one doesn't seem to have the same pattern as before with the 0's until the next block.. :/ --- case d -------------------------- ubuntu@plana48:~$ xfs_bmap /var/lib/ceph/osd/ceph-0/current/3.0_head/DIR_3/plana3922999-382__head_8D203B03__3 /var/lib/ceph/osd/ceph-0/current/3.0_head/DIR_3/plana3922999-382__head_8D203B03__3: 0: [0..7]: 732632192..732632199 1: [8..783]: hole 2: [784..2943]: 733513808..733515967 3: [2944..3511]: hole 4: [3512..4703]: 733516536..733517727 file existed, so these same writes happened prior to the crash and were replayed by the ceph journal. 2013-06-07 17:52:27.927224 7f95cfec0780 15 filestore(/var/lib/ceph/osd/ceph-0) write 3.0_head/8d203b03/plana3922999-382/head//3 0~57 2013-06-07 17:52:27.927291 7f95cfec0780 10 filestore(/var/lib/ceph/osd/ceph-0) file did exist 2013-06-07 17:52:27.927358 7f95cfec0780 10 filestore(/var/lib/ceph/osd/ceph-0) before: size 2404703 blocks 3360 2013-06-07 17:52:27.927367 7f95cfec0780 20 filestore(/var/lib/ceph/osd/ceph-0) write lseek fd 100 2013-06-07 17:52:27.927565 7f95cfec0780 10 filestore(/var/lib/ceph/osd/ceph-0) after: size 2404703 blocks 3360 2013-06-07 17:52:27.927586 7f95cfec0780 10 filestore(/var/lib/ceph/osd/ceph-0) write 3.0_head/8d203b03/plana3922999-382/head//3 0~57 = 57 2013-06-07 17:52:27.945189 7f95cfec0780 15 filestore(/var/lib/ceph/osd/ceph-0) write 3.0_head/8d203b03/plana3922999-382/head//3 404703~773518 2013-06-07 17:52:27.945240 7f95cfec0780 10 filestore(/var/lib/ceph/osd/ceph-0) file did exist 2013-06-07 17:52:27.945310 7f95cfec0780 10 filestore(/var/lib/ceph/osd/ceph-0) before: size 2404703 blocks 3360 2013-06-07 17:52:27.945319 7f95cfec0780 20 filestore(/var/lib/ceph/osd/ceph-0) write lseek fd 100 2013-06-07 17:52:27.960204 7f95cfec0780 10 filestore(/var/lib/ceph/osd/ceph-0) after: size 2404703 blocks 3360 2013-06-07 17:52:27.960225 7f95cfec0780 10 filestore(/var/lib/ceph/osd/ceph-0) write 3.0_head/8d203b03/plana3922999-382/head//3 404703~773518 = 773518 2013-06-07 17:52:27.982497 7f95cfec0780 15 filestore(/var/lib/ceph/osd/ceph-0) write 3.0_head/8d203b03/plana3922999-382/head//3 1801584~603119 2013-06-07 17:52:27.982550 7f95cfec0780 10 filestore(/var/lib/ceph/osd/ceph-0) file did exist 2013-06-07 17:52:27.982607 7f95cfec0780 10 filestore(/var/lib/ceph/osd/ceph-0) before: size 2404703 blocks 3360 2013-06-07 17:52:27.982615 7f95cfec0780 20 filestore(/var/lib/ceph/osd/ceph-0) write lseek fd 101 2013-06-07 17:52:28.005284 7f95cfec0780 10 filestore(/var/lib/ceph/osd/ceph-0) after: size 2404703 blocks 3360 2013-06-07 17:52:28.005305 7f95cfec0780 10 filestore(/var/lib/ceph/osd/ceph-0) write 3.0_head/8d203b03/plana3922999-382/head//3 1801584~603119 = 603119 INFO:teuthology.task.rados.rados.0.out:incorrect buffer at pos 1179648 (((same offset as previous case!!!)) 0011fa10 dc 59 b9 d9 45 6b d8 5b 7d e2 e7 fa e8 5d 09 bb |.Y..Ek.[}....]..| 0011fa20 7b 20 72 2f ad 92 ba d6 2e 88 f4 49 59 43 81 16 |{ r/.......IYC..| 0011fa30 69 39 a3 69 6e 96 d3 6f fd 80 d2 7a a8 1d 81 e3 |i9.in..o...z....| 0011fa40 33 55 71 b2 83 56 38 fe 55 db 89 15 b3 2c 01 5b |3Uq..V8.U....,.[| 0011fa50 26 e4 c1 f2 ab 74 bf 1d 65 6a dc 64 17 71 b4 76 |&....t..ej.d.q.v| 0011fa60 4f 17 38 d2 61 51 fc 24 1a bf 50 6f 30 00 00 00 |O.8.aQ.$..Po0...| <-- correct end to earlier write 0011fa70 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00120000 32 2c 24 20 ba c7 ae 6f 10 c3 a1 86 75 61 84 71 |2,$ ...o....ua.q| <-- garbage starting here 00120010 9d 34 d7 19 6d ad b4 3e 81 75 3a 28 6f 7d a9 17 |.4..m..>.u:(o}..| 00120020 cd d6 4b 7e f2 40 19 f3 7f 83 19 e3 9c 80 87 08 |..K~.@..........| 00120030 02 06 bb 9b bb 65 c4 49 ab c3 d7 e6 cd 30 6a e2 |.....e.I.....0j.| 00120040 3d 7a 1e 7f f9 c0 60 be 66 c8 ce 1d 93 11 59 02 |=z....`.f.....Y.| 00120050 3f a7 2f f5 9c b8 55 90 d0 e9 16 36 3f 68 1e 85 |?./...U....6?h..| ... 0012f810 b2 fa 86 ee e4 18 b2 94 d3 5c 3e 16 93 39 eb 35 |.........\>..9.5| 0012f820 3b 3e 84 e5 a4 ac 80 3b 00 00 00 00 00 00 00 00 |;>.....;........| 0012f830 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| to here * 001b7d70 2d c2 02 5b d8 37 bf 54 ad 1d f8 1a e8 35 d8 c5 |-..[.7.T.....5..| <-- this is the next write.. 001b7d80 39 44 7d 57 b1 93 4b 23 7a 39 fd f3 49 1a 80 e7 |9D}W..K#z9..I...| 001b7d90 93 ad 7f 1d 80 80 bc 16 a3 7d 24 d2 66 e9 9f c0 |.........}$.f...| this pattern looks just like the ones i saw previously.