2019-05-20 19:08:12 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-09-21 15:37:46 +10:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-09-30 14:23:59 +10:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2009-04-14 12:01:53 +10:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2009-04-14 12:01:53 +10:00
|
|
|
|
2015-05-22 17:13:32 -04:00
|
|
|
|
2015-12-24 19:20:34 -07:00
|
|
|
|
2009-04-14 12:01:53 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-15 17:22:21 +08:00
|
|
|
|
2024-03-03 07:01:40 -07:00
|
|
|
|
[PATCH] BLOCK: Make it possible to disable the block layer [try #6]
Make it possible to disable the block layer. Not all embedded devices require
it, some can make do with just JFFS2, NFS, ramfs, etc - none of which require
the block layer to be present.
This patch does the following:
(*) Introduces CONFIG_BLOCK to disable the block layer, buffering and blockdev
support.
(*) Adds dependencies on CONFIG_BLOCK to any configuration item that controls
an item that uses the block layer. This includes:
(*) Block I/O tracing.
(*) Disk partition code.
(*) All filesystems that are block based, eg: Ext3, ReiserFS, ISOFS.
(*) The SCSI layer. As far as I can tell, even SCSI chardevs use the
block layer to do scheduling. Some drivers that use SCSI facilities -
such as USB storage - end up disabled indirectly from this.
(*) Various block-based device drivers, such as IDE and the old CDROM
drivers.
(*) MTD blockdev handling and FTL.
(*) JFFS - which uses set_bdev_super(), something it could avoid doing by
taking a leaf out of JFFS2's book.
(*) Makes most of the contents of linux/blkdev.h, linux/buffer_head.h and
linux/elevator.h contingent on CONFIG_BLOCK being set. sector_div() is,
however, still used in places, and so is still available.
(*) Also made contingent are the contents of linux/mpage.h, linux/genhd.h and
parts of linux/fs.h.
(*) Makes a number of files in fs/ contingent on CONFIG_BLOCK.
(*) Makes mm/bounce.c (bounce buffering) contingent on CONFIG_BLOCK.
(*) set_page_dirty() doesn't call __set_page_dirty_buffers() if CONFIG_BLOCK
is not enabled.
(*) fs/no-block.c is created to hold out-of-line stubs and things that are
required when CONFIG_BLOCK is not set:
(*) Default blockdev file operations (to give error ENODEV on opening).
(*) Makes some /proc changes:
(*) /proc/devices does not list any blockdevs.
(*) /proc/diskstats and /proc/partitions are contingent on CONFIG_BLOCK.
(*) Makes some compat ioctl handling contingent on CONFIG_BLOCK.
(*) If CONFIG_BLOCK is not defined, makes sys_quotactl() return -ENODEV if
given command other than Q_SYNC or if a special device is specified.
(*) In init/do_mounts.c, no reference is made to the blockdev routines if
CONFIG_BLOCK is not defined. This does not prohibit NFS roots or JFFS2.
(*) The bdflush, ioprio_set and ioprio_get syscalls can now be absent (return
error ENOSYS by way of cond_syscall if so).
(*) The seclvl_bd_claim() and seclvl_bd_release() security calls do nothing if
CONFIG_BLOCK is not set, since they can't then happen.
Signed-Off-By: David Howells <dhowells@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2006-09-30 20:45:40 +02:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2026-01-15 01:12:33 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2025-02-15 17:22:21 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-08-29 16:04:26 +08:00
|
|
|
|
2025-02-15 17:22:21 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-07-07 09:27:11 +08:00
|
|
|
|
2025-08-29 16:04:26 +08:00
|
|
|
|
2025-08-29 16:04:19 +08:00
|
|
|
|
2025-02-15 17:22:21 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-11-18 16:16:11 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-12-23 10:49:00 +01:00
|
|
|
|
2024-06-11 21:22:41 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-12-23 10:49:00 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-10-11 16:45:26 +11:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2009-03-31 14:33:13 +11:00
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
md: fix is_mddev_idle()
If sync_speed is above speed_min, then is_mddev_idle() will be called
for each sync IO to check if the array is idle, and inflight sync_io
will be limited if the array is not idle.
However, while mkfs.ext4 for a large raid5 array while recovery is in
progress, it's found that sync_speed is already above speed_min while
lots of stripes are used for sync IO, causing long delay for mkfs.ext4.
Root cause is the following checking from is_mddev_idle():
t1: submit sync IO: events1 = completed IO - issued sync IO
t2: submit next sync IO: events2 = completed IO - issued sync IO
if (events2 - events1 > 64)
For consequence, the more sync IO issued, the less likely checking will
pass. And when completed normal IO is more than issued sync IO, the
condition will finally pass and is_mddev_idle() will return false,
however, last_events will be updated hence is_mddev_idle() can only
return false once in a while.
Fix this problem by changing the checking as following:
1) mddev doesn't have normal IO completed;
2) mddev doesn't have normal IO inflight;
3) if any member disks is partition, and all other partitions doesn't
have IO completed.
Also change rdev->last_events to unsigned long to cleanup type casting.
Link: https://lore.kernel.org/linux-raid/20250506124903.2540268-9-yukuai1@huaweicloud.com
Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Reviewed-by: Xiao Ni <xni@redhat.com>
2025-05-06 20:49:02 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-01-14 09:14:34 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2024-01-23 14:26:21 +01:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-07-28 11:31:47 +10:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2006-06-26 00:27:57 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2012-05-21 09:27:00 +10:00
|
|
|
|
2014-09-30 14:23:59 +10:00
|
|
|
|
2005-09-09 16:23:53 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2005-11-08 21:39:24 -08:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-12-23 10:17:51 +11:00
|
|
|
|
2008-04-30 00:52:32 -07:00
|
|
|
|
2005-09-09 16:23:45 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2010-06-15 09:36:03 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-06-21 17:17:25 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-08-13 14:31:56 -07:00
|
|
|
|
|
|
|
|
|
2006-06-26 00:27:40 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-08-13 14:31:56 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-11-08 21:39:31 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-06-17 17:33:10 +02:00
|
|
|
|
2009-12-14 12:49:58 +11:00
|
|
|
|
|
|
|
|
|
2006-01-06 00:20:52 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-06-19 17:30:46 +08:00
|
|
|
|
2019-12-23 10:49:00 +01:00
|
|
|
|
2019-06-19 17:30:46 +08:00
|
|
|
|
2013-12-11 14:11:53 -05:00
|
|
|
|
2008-10-21 13:25:28 +11:00
|
|
|
|
2020-07-14 16:10:26 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-12-24 19:20:34 -07:00
|
|
|
|
2017-03-09 09:59:57 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-12-23 10:17:51 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-12-12 10:13:33 +11:00
|
|
|
|
2017-07-04 11:20:30 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2013-12-12 10:13:33 +11:00
|
|
|
|
2011-12-23 10:17:51 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-10-29 18:51:31 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-08-13 14:31:55 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-12-21 10:50:59 +11:00
|
|
|
|
2016-10-21 16:26:57 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-11-18 16:16:11 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-11-18 16:16:11 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-12-23 10:48:53 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-06-19 17:30:46 +08:00
|
|
|
|
2024-02-29 17:57:06 +08:00
|
|
|
|
2011-12-23 10:17:51 +11:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2025-02-27 15:55:07 +08:00
|
|
|
|
|
|
|
|
|
2011-07-28 11:31:46 +10:00
|
|
|
|
|
|
|
|
|
2015-12-24 19:20:34 -07:00
|
|
|
|
2011-07-28 11:31:46 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-02-29 17:57:04 +08:00
|
|
|
|
|
|
|
|
|
2026-07-10 06:23:29 -07:00
|
|
|
|
2024-02-29 17:57:04 +08:00
|
|
|
|
|
|
|
|
|
2025-02-27 15:55:07 +08:00
|
|
|
|
2024-02-29 17:57:04 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-10 06:23:29 -07:00
|
|
|
|
2025-02-27 15:55:06 +08:00
|
|
|
|
2026-07-10 06:23:29 -07:00
|
|
|
|
2025-02-27 15:55:06 +08:00
|
|
|
|
2014-03-29 10:20:02 -05:00
|
|
|
|
2025-02-15 17:22:24 +08:00
|
|
|
|
2014-03-29 10:20:02 -05:00
|
|
|
|
2022-03-22 16:23:38 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2022-07-19 11:18:23 +02:00
|
|
|
|
2026-01-15 01:12:30 +08:00
|
|
|
|
2026-01-15 01:12:31 +08:00
|
|
|
|
2026-01-15 01:12:32 +08:00
|
|
|
|
2026-04-28 19:20:10 -04:00
|
|
|
|
2022-03-22 16:23:38 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-11-09 10:21:33 +11:00
|
|
|
|
2022-03-22 16:23:38 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-10-29 14:34:19 +08:00
|
|
|
|
2022-07-19 11:18:23 +02:00
|
|
|
|
2026-01-15 01:12:30 +08:00
|
|
|
|
2026-01-15 01:12:31 +08:00
|
|
|
|
2026-01-15 01:12:32 +08:00
|
|
|
|
2026-04-28 19:20:10 -04:00
|
|
|
|
2016-11-09 10:21:33 +11:00
|
|
|
|
2016-12-08 15:48:19 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-12-23 10:48:53 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-12-23 10:49:00 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
md/raid1: serialize overlap io for writemostly disk
Previously, using wait_event() would wake up all waiters simultaneously,
and they would compete for the tree lock. The bio which gets the lock
first will be handled, so the write sequence cannot be guaranteed.
For example:
bio1(100,200)
bio2(150,200)
bio3(150,300)
The write sequence of fast device is bio1,bio2,bio3. But the write sequence
of slow device could be bio1,bio3,bio2 due to lock competition. This causes
data corruption.
Replace waitqueue with a fifo list to guarantee the write sequence. And it
also needs to iterate the list when removing one entry. If not, it may miss
the opportunity to wake up the waiting io.
For example:
bio1(1,3), bio2(2,4)
bio3(5,7), bio4(6,8)
These four bios are in the same bucket. bio1 and bio3 are inserted into
the rbtree. bio2 and bio4 are added to the waiting list and bio2 is the
first one. bio3 returns from slow disk and tries to wake up the waiting
bios. bio2 is removed from the list and will be handled. But bio1 hasn't
finished. So bio2 will be added into waiting list again. Then bio1 returns
from slow disk and wakes up waiting bios. bio4 is removed from the list
and will be handled. Now bio1, bio3 and bio4 all finish and bio2 is left
on the waiting list. So it needs to iterate the waiting list to wake up
the right bio.
Signed-off-by: Xiao Ni <xni@redhat.com>
Link: https://lore.kernel.org/linux-raid/20260324072501.59865-1-xni@redhat.com/
Signed-off-by: Yu Kuai <yukuai@fnnas.com>
2026-03-24 15:24:54 +08:00
|
|
|
|
2019-12-23 10:49:00 +01:00
|
|
|
|
md/raid1: serialize overlap io for writemostly disk
Previously, using wait_event() would wake up all waiters simultaneously,
and they would compete for the tree lock. The bio which gets the lock
first will be handled, so the write sequence cannot be guaranteed.
For example:
bio1(100,200)
bio2(150,200)
bio3(150,300)
The write sequence of fast device is bio1,bio2,bio3. But the write sequence
of slow device could be bio1,bio3,bio2 due to lock competition. This causes
data corruption.
Replace waitqueue with a fifo list to guarantee the write sequence. And it
also needs to iterate the list when removing one entry. If not, it may miss
the opportunity to wake up the waiting io.
For example:
bio1(1,3), bio2(2,4)
bio3(5,7), bio4(6,8)
These four bios are in the same bucket. bio1 and bio3 are inserted into
the rbtree. bio2 and bio4 are added to the waiting list and bio2 is the
first one. bio3 returns from slow disk and tries to wake up the waiting
bios. bio2 is removed from the list and will be handled. But bio1 hasn't
finished. So bio2 will be added into waiting list again. Then bio1 returns
from slow disk and wakes up waiting bios. bio4 is removed from the list
and will be handled. Now bio1, bio3 and bio4 all finish and bio2 is left
on the waiting list. So it needs to iterate the waiting list to wake up
the right bio.
Signed-off-by: Xiao Ni <xni@redhat.com>
Link: https://lore.kernel.org/linux-raid/20260324072501.59865-1-xni@redhat.com/
Signed-off-by: Yu Kuai <yukuai@fnnas.com>
2026-03-24 15:24:54 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2019-06-19 17:30:46 +08:00
|
|
|
|
|
|
|
|
|
2022-06-08 10:27:54 -06:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-10-11 16:49:58 +11:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2013-08-27 16:28:23 +10:00
|
|
|
|
2006-10-03 01:15:46 -07:00
|
|
|
|
2016-12-08 15:48:19 -08:00
|
|
|
|
2006-10-03 01:15:46 -07:00
|
|
|
|
2009-03-31 14:39:39 +11:00
|
|
|
|
2023-10-10 23:19:43 +08:00
|
|
|
|
2023-01-31 13:17:10 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-08-08 21:18:03 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-05-06 20:49:00 +08:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2005-11-08 21:39:23 -08:00
|
|
|
|
md: make devices disappear when they are no longer needed.
Currently md devices, once created, never disappear until the module
is unloaded. This is essentially because the gendisk holds a
reference to the mddev, and the mddev holds a reference to the
gendisk, this a circular reference.
If we drop the reference from mddev to gendisk, then we need to ensure
that the mddev is destroyed when the gendisk is destroyed. However it
is not possible to hook into the gendisk destruction process to enable
this.
So we drop the reference from the gendisk to the mddev and destroy the
gendisk when the mddev gets destroyed. However this has a
complication.
Between the call
__blkdev_get->get_gendisk->kobj_lookup->md_probe
and the call
__blkdev_get->md_open
there is no obvious way to hold a reference on the mddev any more, so
unless something is done, it will disappear and gendisk will be
destroyed prematurely.
Also, once we decide to destroy the mddev, there will be an unlockable
moment before the gendisk is unlinked (blk_unregister_region) during
which a new reference to the gendisk can be created. We need to
ensure that this reference can not be used. i.e. the ->open must
fail.
So:
1/ in md_probe we set a flag in the mddev (hold_active) which
indicates that the array should be treated as active, even
though there are no references, and no appearance of activity.
This is cleared by md_release when the device is closed if it
is no longer needed.
This ensures that the gendisk will survive between md_probe and
md_open.
2/ In md_open we check if the mddev we expect to open matches
the gendisk that we did open.
If there is a mismatch we return -ERESTARTSYS and modify
__blkdev_get to retry from the top in that case.
In the -ERESTARTSYS sys case we make sure to wait until
the old gendisk (that we succeeded in opening) is really gone so
we loop at most once.
Some udev configurations will always open an md device when it first
appears. If we allow an md device that was just created by an open
to disappear on an immediate close, then this can race with such udev
configurations and result in an infinite loop the device being opened
and closed, then re-open due to the 'ADD' even from the first open,
and then close and so on.
So we make sure an md device, once created by an open, remains active
at least until some md 'ioctl' has been made on it. This means that
all normal usage of md devices will allow them to disappear promptly
when not needed, but the worst that an incorrect usage will do it
cause an inactive md device to be left in existence (it can easily be
removed).
As an array can be stopped by writing to a sysfs attribute
echo clear > /sys/block/mdXXX/md/array_state
we need to use scheduled work for deleting the gendisk and other
kobjects. This allows us to wait for any pending gendisk deletion to
complete by simply calling flush_scheduled_work().
Signed-off-by: NeilBrown <neilb@suse.de>
2009-01-09 08:31:10 +11:00
|
|
|
|
|
|
|
|
|
2009-01-09 08:31:10 +11:00
|
|
|
|
2005-11-08 21:39:23 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-09-30 14:23:59 +10:00
|
|
|
|
2008-02-06 01:39:51 -08:00
|
|
|
|
|
|
|
|
|
2020-12-14 12:08:48 -05:00
|
|
|
|
2015-12-21 10:51:01 +11:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2006-01-06 00:20:51 -08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2014-09-30 14:23:59 +10:00
|
|
|
|
2009-03-31 14:33:13 +11:00
|
|
|
|
2008-07-21 17:05:22 +10:00
|
|
|
|
2009-03-31 15:00:31 +11:00
|
|
|
|
|
|
|
|
|
2025-11-03 20:57:57 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2010-05-18 09:28:43 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-03-27 01:18:11 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-06-18 08:45:27 +10:00
|
|
|
|
2020-12-14 12:08:48 -05:00
|
|
|
|
2012-05-21 09:27:00 +10:00
|
|
|
|
2006-03-27 01:18:11 -08:00
|
|
|
|
2023-05-23 10:10:17 +08:00
|
|
|
|
|
|
|
|
|
2013-06-25 01:23:59 -05:00
|
|
|
|
2024-06-11 21:22:48 +08:00
|
|
|
|
|
|
|
|
|
2013-06-25 01:23:59 -05:00
|
|
|
|
2024-06-11 21:22:48 +08:00
|
|
|
|
2013-06-25 01:23:59 -05:00
|
|
|
|
2024-06-11 21:22:48 +08:00
|
|
|
|
2006-07-10 04:44:16 -07:00
|
|
|
|
2009-03-31 14:33:13 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2006-07-10 04:44:16 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2005-11-08 21:39:26 -08:00
|
|
|
|
2012-10-11 14:17:59 +11:00
|
|
|
|
2005-11-08 21:39:26 -08:00
|
|
|
|
|
|
|
|
|
2006-03-27 01:18:14 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-01-06 00:21:36 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-05-06 20:49:01 +08:00
|
|
|
|
2006-01-06 00:21:36 -08:00
|
|
|
|
2008-05-23 13:04:38 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-01-06 00:20:15 -08:00
|
|
|
|
2006-06-26 00:27:40 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-08-10 12:50:52 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-05-25 08:12:56 +02:00
|
|
|
|
2009-08-10 12:50:52 +10:00
|
|
|
|
2021-05-25 08:12:56 +02:00
|
|
|
|
|
|
|
|
|
2009-08-10 12:50:52 +10:00
|
|
|
|
|
|
|
|
|
2006-03-27 01:18:20 -08:00
|
|
|
|
2008-07-21 17:05:25 +10:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-02-24 17:26:41 +11:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
md: fix is_mddev_idle()
If sync_speed is above speed_min, then is_mddev_idle() will be called
for each sync IO to check if the array is idle, and inflight sync_io
will be limited if the array is not idle.
However, while mkfs.ext4 for a large raid5 array while recovery is in
progress, it's found that sync_speed is already above speed_min while
lots of stripes are used for sync IO, causing long delay for mkfs.ext4.
Root cause is the following checking from is_mddev_idle():
t1: submit sync IO: events1 = completed IO - issued sync IO
t2: submit next sync IO: events2 = completed IO - issued sync IO
if (events2 - events1 > 64)
For consequence, the more sync IO issued, the less likely checking will
pass. And when completed normal IO is more than issued sync IO, the
condition will finally pass and is_mddev_idle() will return false,
however, last_events will be updated hence is_mddev_idle() can only
return false once in a while.
Fix this problem by changing the checking as following:
1) mddev doesn't have normal IO completed;
2) mddev doesn't have normal IO inflight;
3) if any member disks is partition, and all other partitions doesn't
have IO completed.
Also change rdev->last_events to unsigned long to cleanup type casting.
Link: https://lore.kernel.org/linux-raid/20250506124903.2540268-9-yukuai1@huaweicloud.com
Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Reviewed-by: Xiao Ni <xni@redhat.com>
2025-05-06 20:49:02 +08:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2025-07-22 11:33:40 +08:00
|
|
|
|
2008-06-28 08:31:24 +10:00
|
|
|
|
|
|
|
|
|
2008-02-06 01:39:52 -08:00
|
|
|
|
|
|
|
|
|
2005-06-21 17:17:12 -07:00
|
|
|
|
2013-12-11 14:11:53 -05:00
|
|
|
|
2008-10-21 13:25:21 +11:00
|
|
|
|
|
|
|
|
|
2013-12-11 14:11:53 -05:00
|
|
|
|
2020-07-14 16:10:26 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2008-10-21 13:25:21 +11:00
|
|
|
|
2023-08-25 11:16:16 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
md: make devices disappear when they are no longer needed.
Currently md devices, once created, never disappear until the module
is unloaded. This is essentially because the gendisk holds a
reference to the mddev, and the mddev holds a reference to the
gendisk, this a circular reference.
If we drop the reference from mddev to gendisk, then we need to ensure
that the mddev is destroyed when the gendisk is destroyed. However it
is not possible to hook into the gendisk destruction process to enable
this.
So we drop the reference from the gendisk to the mddev and destroy the
gendisk when the mddev gets destroyed. However this has a
complication.
Between the call
__blkdev_get->get_gendisk->kobj_lookup->md_probe
and the call
__blkdev_get->md_open
there is no obvious way to hold a reference on the mddev any more, so
unless something is done, it will disappear and gendisk will be
destroyed prematurely.
Also, once we decide to destroy the mddev, there will be an unlockable
moment before the gendisk is unlinked (blk_unregister_region) during
which a new reference to the gendisk can be created. We need to
ensure that this reference can not be used. i.e. the ->open must
fail.
So:
1/ in md_probe we set a flag in the mddev (hold_active) which
indicates that the array should be treated as active, even
though there are no references, and no appearance of activity.
This is cleared by md_release when the device is closed if it
is no longer needed.
This ensures that the gendisk will survive between md_probe and
md_open.
2/ In md_open we check if the mddev we expect to open matches
the gendisk that we did open.
If there is a mismatch we return -ERESTARTSYS and modify
__blkdev_get to retry from the top in that case.
In the -ERESTARTSYS sys case we make sure to wait until
the old gendisk (that we succeeded in opening) is really gone so
we loop at most once.
Some udev configurations will always open an md device when it first
appears. If we allow an md device that was just created by an open
to disappear on an immediate close, then this can race with such udev
configurations and result in an infinite loop the device being opened
and closed, then re-open due to the 'ADD' even from the first open,
and then close and so on.
So we make sure an md device, once created by an open, remains active
at least until some md 'ioctl' has been made on it. This means that
all normal usage of md devices will allow them to disappear promptly
when not needed, but the worst that an incorrect usage will do it
cause an inactive md device to be left in existence (it can easily be
removed).
As an array can be stopped by writing to a sysfs attribute
echo clear > /sys/block/mdXXX/md/array_state
we need to use scheduled work for deleting the gendisk and other
kobjects. This allows us to wait for any pending gendisk deletion to
complete by simply calling flush_scheduled_work().
Signed-off-by: NeilBrown <neilb@suse.de>
2009-01-09 08:31:10 +11:00
|
|
|
|
2014-12-15 12:56:56 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-12-15 12:56:58 +11:00
|
|
|
|
2014-12-15 12:56:58 +11:00
|
|
|
|
2014-12-15 12:57:00 +11:00
|
|
|
|
2014-12-15 12:57:01 +11:00
|
|
|
|
|
|
|
|
|
2014-12-15 12:56:56 +11:00
|
|
|
|
|
|
|
|
|
2005-06-21 17:17:26 -07:00
|
|
|
|
2005-06-21 17:17:28 -07:00
|
|
|
|
2005-06-21 17:17:12 -07:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2014-09-30 14:23:59 +10:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
MD: use per-cpu counter for writes_pending
The 'writes_pending' counter is used to determine when the
array is stable so that it can be marked in the superblock
as "Clean". Consequently it needs to be updated frequently
but only checked for zero occasionally. Recent changes to
raid5 cause the count to be updated even more often - once
per 4K rather than once per bio. This provided
justification for making the updates more efficient.
So we replace the atomic counter a percpu-refcount.
This can be incremented and decremented cheaply most of the
time, and can be switched to "atomic" mode when more
precise counting is needed. As it is possible for multiple
threads to want a precise count, we introduce a
"sync_checker" counter to count the number of threads
in "set_in_sync()", and only switch the refcount back
to percpu mode when that is zero.
We need to be careful about races between set_in_sync()
setting ->in_sync to 1, and md_write_start() setting it
to zero. md_write_start() holds the rcu_read_lock()
while checking if the refcount is in percpu mode. If
it is, then we know a switch to 'atomic' will not happen until
after we call rcu_read_unlock(), in which case set_in_sync()
will see the elevated count, and not set in_sync to 1.
If it is not in percpu mode, we take the mddev->lock to
ensure proper synchronization.
It is no longer possible to quickly check if the count is zero, which
we previously did to update a timer or to schedule the md_thread.
So now we do these every time we decrement that counter, but make
sure they are fast.
mod_timer() already optimizes the case where the timeout value doesn't
actually change. We leverage that further by always rounding off the
jiffies to the timeout value. This may delay the marking of 'clean'
slightly, but ensure we only perform atomic operation here when absolutely
needed.
md_wakeup_thread() current always calls wake_up(), even if
THREAD_WAKEUP is already set. That too can be optimised to avoid
calls to wake_up().
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Shaohua Li <shli@fb.com>
2017-03-15 14:05:14 +11:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2025-08-29 16:04:19 +08:00
|
|
|
|
2024-08-26 15:44:52 +08:00
|
|
|
|
2024-08-26 15:44:21 +08:00
|
|
|
|
2009-12-14 12:49:52 +11:00
|
|
|
|
|
|
|
|
|
2009-12-14 12:49:54 +11:00
|
|
|
|
2009-12-14 12:49:52 +11:00
|
|
|
|
|
|
|
|
|
2009-12-14 12:49:54 +11:00
|
|
|
|
2014-09-30 14:23:59 +10:00
|
|
|
|
2009-12-14 12:49:52 +11:00
|
|
|
|
2012-05-22 13:55:07 +10:00
|
|
|
|
2009-12-14 12:49:54 +11:00
|
|
|
|
2009-12-14 12:49:52 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-05-22 13:55:07 +10:00
|
|
|
|
|
|
|
|
|
2009-12-14 12:49:52 +11:00
|
|
|
|
2009-12-14 12:49:53 +11:00
|
|
|
|
2010-06-01 19:37:31 +10:00
|
|
|
|
2009-12-14 12:49:53 +11:00
|
|
|
|
2009-12-14 12:49:56 +11:00
|
|
|
|
2014-03-29 10:20:02 -05:00
|
|
|
|
2014-03-30 00:42:49 -05:00
|
|
|
|
2009-12-14 12:49:52 +11:00
|
|
|
|
2005-06-21 17:17:14 -07:00
|
|
|
|
2014-09-30 14:23:59 +10:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2009-12-14 12:49:49 +11:00
|
|
|
|
2021-05-29 12:30:49 +02:00
|
|
|
|
2010-06-01 19:37:29 +10:00
|
|
|
|
2018-05-20 18:25:52 -04:00
|
|
|
|
|
|
|
|
|
2017-06-21 09:12:21 +10:00
|
|
|
|
|
|
|
|
|
2023-06-22 00:51:04 +08:00
|
|
|
|
2010-10-26 18:31:13 +11:00
|
|
|
|
2010-07-26 11:49:55 +10:00
|
|
|
|
2019-12-23 10:48:53 +01:00
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
2014-03-29 10:20:02 -05:00
|
|
|
|
2025-02-15 17:22:25 +08:00
|
|
|
|
2015-12-21 10:51:00 +11:00
|
|
|
|
2018-02-02 23:13:19 +01:00
|
|
|
|
md: fix duplicate filename for rdev
Commit 5792a2856a63 ("[PATCH] md: avoid a deadlock when removing a device
from an md array via sysfs") delays the deletion of rdev, however, this
introduces a window that rdev can be added again while the deletion is
not done yet, and sysfs will complain about duplicate filename.
Follow up patches try to fix this problem by flushing workqueue, however,
flush_rdev_wq() is just dead code, the progress in
md_kick_rdev_from_array():
1) list_del_rcu(&rdev->same_set);
2) synchronize_rcu();
3) queue_work(md_rdev_misc_wq, &rdev->del_work);
So in flush_rdev_wq(), if rdev is found in the list, work_pending() can
never pass, in the meantime, if work is queued, then rdev can never be
found in the list.
flush_rdev_wq() can be replaced by flush_workqueue() directly, however,
this approach is not good:
- the workqueue is global, this synchronization for all raid disks is
not necessary.
- flush_workqueue can't be called under 'reconfig_mutex', there is still
a small window between flush_workqueue() and mddev_lock() that other
contexts can queue new work, hence the problem is not solved completely.
sysfs already has apis to support delete itself through writer, and
these apis, specifically sysfs_break/unbreak_active_protection(), is used
to support deleting rdev synchronously. Therefore, the above commit can be
reverted, and sysfs duplicate filename can be avoided.
A new mdadm regression test is proposed as well([1]).
[1] https://lore.kernel.org/linux-raid/20230428062845.1975462-1-yukuai1@huaweicloud.com/
Fixes: 5792a2856a63 ("[PATCH] md: avoid a deadlock when removing a device from an md array via sysfs")
Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Signed-off-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/r/20230523012727.3042247-1-yukuai1@huaweicloud.com
2023-05-23 09:27:27 +08:00
|
|
|
|
|
|
|
|
|
2023-06-21 22:29:33 +08:00
|
|
|
|
md: fix duplicate filename for rdev
Commit 5792a2856a63 ("[PATCH] md: avoid a deadlock when removing a device
from an md array via sysfs") delays the deletion of rdev, however, this
introduces a window that rdev can be added again while the deletion is
not done yet, and sysfs will complain about duplicate filename.
Follow up patches try to fix this problem by flushing workqueue, however,
flush_rdev_wq() is just dead code, the progress in
md_kick_rdev_from_array():
1) list_del_rcu(&rdev->same_set);
2) synchronize_rcu();
3) queue_work(md_rdev_misc_wq, &rdev->del_work);
So in flush_rdev_wq(), if rdev is found in the list, work_pending() can
never pass, in the meantime, if work is queued, then rdev can never be
found in the list.
flush_rdev_wq() can be replaced by flush_workqueue() directly, however,
this approach is not good:
- the workqueue is global, this synchronization for all raid disks is
not necessary.
- flush_workqueue can't be called under 'reconfig_mutex', there is still
a small window between flush_workqueue() and mddev_lock() that other
contexts can queue new work, hence the problem is not solved completely.
sysfs already has apis to support delete itself through writer, and
these apis, specifically sysfs_break/unbreak_active_protection(), is used
to support deleting rdev synchronously. Therefore, the above commit can be
reverted, and sysfs duplicate filename can be avoided.
A new mdadm regression test is proposed as well([1]).
[1] https://lore.kernel.org/linux-raid/20230428062845.1975462-1-yukuai1@huaweicloud.com/
Fixes: 5792a2856a63 ("[PATCH] md: avoid a deadlock when removing a device from an md array via sysfs")
Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Signed-off-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/r/20230523012727.3042247-1-yukuai1@huaweicloud.com
2023-05-23 09:27:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
md: refactor idle/frozen_sync_thread() to fix deadlock
Our test found a following deadlock in raid10:
1) Issue a normal write, and such write failed:
raid10_end_write_request
set_bit(R10BIO_WriteError, &r10_bio->state)
one_write_done
reschedule_retry
// later from md thread
raid10d
handle_write_completed
list_add(&r10_bio->retry_list, &conf->bio_end_io_list)
// later from md thread
raid10d
if (!test_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags))
list_move(conf->bio_end_io_list.prev, &tmp)
r10_bio = list_first_entry(&tmp, struct r10bio, retry_list)
raid_end_bio_io(r10_bio)
Dependency chain 1: normal io is waiting for updating superblock
2) Trigger a recovery:
raid10_sync_request
raise_barrier
Dependency chain 2: sync thread is waiting for normal io
3) echo idle/frozen to sync_action:
action_store
mddev_lock
md_unregister_thread
kthread_stop
Dependency chain 3: drop 'reconfig_mutex' is waiting for sync thread
4) md thread can't update superblock:
raid10d
md_check_recovery
if (mddev_trylock(mddev))
md_update_sb
Dependency chain 4: update superblock is waiting for 'reconfig_mutex'
Hence cyclic dependency exist, in order to fix the problem, we must
break one of them. Dependency 1 and 2 can't be broken because they are
foundation design. Dependency 4 may be possible if it can be guaranteed
that no io can be inflight, however, this requires a new mechanism which
seems complex. Dependency 3 is a good choice, because idle/frozen only
requires sync thread to finish, which can be done asynchronously that is
already implemented, and 'reconfig_mutex' is not needed anymore.
This patch switch 'idle' and 'frozen' to wait sync thread to be done
asynchronously, and this patch also add a sequence counter to record how
many times sync thread is done, so that 'idle' won't keep waiting on new
started sync thread.
Noted that raid456 has similiar deadlock([1]), and it's verified[2] this
deadlock can be fixed by this patch as well.
[1] https://lore.kernel.org/linux-raid/5ed54ffc-ce82-bf66-4eff-390cb23bc1ac@molgen.mpg.de/T/#t
[2] https://lore.kernel.org/linux-raid/e9067438-d713-f5f3-0d3d-9e6b0e9efa0e@huaweicloud.com/
Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Signed-off-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/r/20230529132037.2124527-5-yukuai1@huaweicloud.com
2023-05-29 21:20:35 +08:00
|
|
|
|
|
|
|
|
|
2016-11-09 10:21:33 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-06-11 21:22:40 +08:00
|
|
|
|
|
|
|
|
|
2016-11-09 10:21:33 +11:00
|
|
|
|
2024-06-11 21:22:40 +08:00
|
|
|
|
|
|
|
|
|
2016-11-09 10:21:33 +11:00
|
|
|
|
2024-06-11 21:22:40 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-06-11 21:22:41 +08:00
|
|
|
|
2024-06-11 21:22:40 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-08-29 16:04:24 +08:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2024-03-05 15:23:00 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-03-05 15:23:01 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-12-15 12:57:01 +11:00
|
|
|
|
|
|
|
|
|
2025-06-11 15:31:06 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-12-15 12:57:01 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-06-11 15:31:06 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-12-15 12:57:01 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-06-11 15:31:06 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-27 15:39:27 +08:00
|
|
|
|
|
|
|
|
|
2025-06-11 15:31:06 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-12-15 12:57:01 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-10-11 16:49:58 +11:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2025-02-15 17:22:21 +08:00
|
|
|
|
2025-02-15 17:22:22 +08:00
|
|
|
|
2019-09-16 13:15:14 -04:00
|
|
|
|
2017-11-19 22:17:01 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
2017-11-19 22:17:01 -08:00
|
|
|
|
|
|
|
|
|
2014-12-15 12:56:58 +11:00
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-09-30 14:23:59 +10:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
|
|
|
|
|
2011-12-23 10:17:51 +11:00
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
2024-06-11 21:22:50 +08:00
|
|
|
|
|
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
md-cluster/raid10: support add disk under grow mode
For clustered raid10 scenario, we need to let all the nodes
know about that a new disk is added to the array, and the
reshape caused by add new member just need to be happened in
one node, but other nodes should know about the change.
Since reshape means read data from somewhere (which is already
used by array) and write data to unused region. Obviously, it
is awful if one node is reading data from address while another
node is writing to the same address. Considering we have
implemented suspend writes in the resyncing area, so we can
just broadcast the reading address to other nodes to avoid the
trouble.
For master node, it would call reshape_request then update sb
during the reshape period. To avoid above trouble, we call
resync_info_update to send RESYNC message in reshape_request.
Then from slave node's view, it receives two type messages:
1. RESYNCING message
Slave node add the address (where master node reading data from)
to suspend list.
2. METADATA_UPDATED message
Once slave nodes know the reshaping is started in master node,
it is time to update reshape position and call start_reshape to
follow master node's step. After reshape is done, only reshape
position is need to be updated, so the majority task of reshaping
is happened on the master node.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
Signed-off-by: Shaohua Li <shli@fb.com>
2018-10-18 16:37:42 +08:00
|
|
|
|
2024-03-05 15:23:04 +08:00
|
|
|
|
2017-10-19 12:49:15 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-09-09 16:23:45 -07:00
|
|
|
|
2017-10-19 12:49:15 +11:00
|
|
|
|
2009-03-31 14:39:39 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
2017-03-09 10:00:03 +01:00
|
|
|
|
|
|
|
|
|
2025-01-09 09:51:43 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-08-03 03:50:21 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2005-11-08 21:39:30 -08:00
|
|
|
|
|
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
|
|
|
|
|
2005-11-08 21:39:30 -08:00
|
|
|
|
|
|
|
|
|
2013-12-11 14:11:53 -05:00
|
|
|
|
2010-06-01 19:37:23 +10:00
|
|
|
|
|
|
|
|
|
2013-09-11 23:19:13 -04:00
|
|
|
|
2010-06-01 19:37:23 +10:00
|
|
|
|
|
|
|
|
|
2013-12-11 14:11:53 -05:00
|
|
|
|
2010-06-01 19:37:23 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
2011-07-27 11:00:36 +10:00
|
|
|
|
|
|
|
|
|
2015-12-18 15:19:16 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-12-23 10:17:51 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-07-27 11:00:36 +10:00
|
|
|
|
|
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
2011-07-27 11:00:36 +10:00
|
|
|
|
|
|
|
|
|
2015-12-18 15:19:16 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-12-23 10:17:51 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-07-27 11:00:36 +10:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-09 08:31:08 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-03-19 12:46:39 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2009-01-09 08:31:08 +11:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2008-07-21 17:05:25 +10:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-10-11 16:48:23 +11:00
|
|
|
|
2012-10-11 13:34:00 +11:00
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2014-09-30 14:23:59 +10:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2005-06-21 17:17:14 -07:00
|
|
|
|
2012-10-11 13:34:00 +11:00
|
|
|
|
2011-10-11 16:48:23 +11:00
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
2023-06-22 00:51:04 +08:00
|
|
|
|
2023-02-03 13:13:44 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
md/md-bitmap: move bitmap_{start, end}write to md upper layer
There are two BUG reports that raid5 will hang at
bitmap_startwrite([1],[2]), root cause is that bitmap start write and end
write is unbalanced, it's not quite clear where, and while reviewing raid5
code, it's found that bitmap operations can be optimized. For example,
for a 4 disks raid5, with chunksize=8k, if user issue a IO (0 + 48k) to
the array:
┌────────────────────────────────────────────────────────────┐
│chunk 0 │
│ ┌────────────┬─────────────┬─────────────┬────────────┼
│ sh0 │A0: 0 + 4k │A1: 8k + 4k │A2: 16k + 4k │A3: P │
│ ┼────────────┼─────────────┼─────────────┼────────────┼
│ sh1 │B0: 4k + 4k │B1: 12k + 4k │B2: 20k + 4k │B3: P │
┼──────┴────────────┴─────────────┴─────────────┴────────────┼
│chunk 1 │
│ ┌────────────┬─────────────┬─────────────┬────────────┤
│ sh2 │C0: 24k + 4k│C1: 32k + 4k │C2: P │C3: 40k + 4k│
│ ┼────────────┼─────────────┼─────────────┼────────────┼
│ sh3 │D0: 28k + 4k│D1: 36k + 4k │D2: P │D3: 44k + 4k│
└──────┴────────────┴─────────────┴─────────────┴────────────┘
Before this patch, 4 stripe head will be used, and each sh will attach
bio for 3 disks, and each attached bio will trigger
bitmap_startwrite() once, which means total 12 times.
- 3 times (0 + 4k), for (A0, A1 and A2)
- 3 times (4 + 4k), for (B0, B1 and B2)
- 3 times (8 + 4k), for (C0, C1 and C3)
- 3 times (12 + 4k), for (D0, D1 and D3)
After this patch, md upper layer will calculate that IO range (0 + 48k)
is corresponding to the bitmap (0 + 16k), and call bitmap_startwrite()
just once.
Noted that this patch will align bitmap ranges to the chunks, for example,
if user issue a IO (0 + 4k) to array:
- Before this patch, 1 time (0 + 4k), for A0;
- After this patch, 1 time (0 + 8k) for chunk 0;
Usually, one bitmap bit will represent more than one disk chunk, and this
doesn't have any difference. And even if user really created a array
that one chunk contain multiple bits, the overhead is that more data
will be recovered after power failure.
Also remove STRIPE_BITMAP_PENDING since it's not used anymore.
[1] https://lore.kernel.org/all/CAJpMwyjmHQLvm6zg1cmQErttNNQPDAAXPKM3xgTjMhbfts986Q@mail.gmail.com/
[2] https://lore.kernel.org/all/ADF7D720-5764-4AF3-B68E-1845988737AA@flyingcircus.io/
Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Link: https://lore.kernel.org/r/20250109015145.158868-6-yukuai1@huaweicloud.com
Signed-off-by: Song Liu <song@kernel.org>
2025-01-09 09:51:45 +08:00
|
|
|
|
|
|
|
|
|
2025-08-29 16:04:18 +08:00
|
|
|
|
2023-02-03 13:13:44 +08:00
|
|
|
|
2021-05-25 17:46:17 +08:00
|
|
|
|
|
|
|
|
|
2005-04-16 15:20:36 -07:00
|
|
|
|
|
|
|
|
|
2025-10-15 16:32:27 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2006-01-06 00:20:40 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-15 17:22:21 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-10-11 16:48:23 +11:00
|
|
|
|
2012-10-11 13:34:00 +11:00
|
|
|
|
2011-10-11 16:48:23 +11:00
|
|
|
|
|
|
|
|
|
2023-08-03 15:17:11 +08:00
|
|
|
|
2025-10-15 16:32:27 +08:00
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
2022-06-07 10:03:56 +08:00
|
|
|
|
2024-06-11 21:22:42 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-05-26 02:52:56 +08:00
|
|
|
|
md/raid5: use md_write_start to count stripes, not bios
We use md_write_start() to increase the count of pending writes, and
md_write_end() to decrement the count. We currently count bios
submitted to md/raid5. Change it count stripe_heads that a WRITE bio
has been attached to.
So now, raid5_make_request() calls md_write_start() and then
md_write_end() to keep the count elevated during the setup of the
request.
add_stripe_bio() calls md_write_start() for each stripe_head, and the
completion routines always call md_write_end(), instead of only
calling it when raid5_dec_bi_active_stripes() returns 0.
make_discard_request also calls md_write_start/end().
The parallel between md_write_{start,end} and use of bi_phys_segments
can be seen in that:
Whenever we set bi_phys_segments to 1, we now call md_write_start.
Whenever we increment it on non-read requests with
raid5_inc_bi_active_stripes(), we now call md_write_start().
Whenever we decrement bi_phys_segments on non-read requsts with
raid5_dec_bi_active_stripes(), we now call md_write_end().
This reduces our dependence on keeping a per-bio count of active
stripes in bi_phys_segments.
md_write_inc() is added which parallels md_write_start(), but requires
that a write has already been started, and is certain never to sleep.
This can be used inside a spinlocked region when adding to a write
request.
Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: Shaohua Li <shli@fb.com>
2017-03-15 14:05:12 +11:00
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
2026-01-05 19:02:50 +08:00
|
|
|
|
|
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
2012-05-21 09:27:00 +10:00
|
|
|
|
2021-02-04 15:50:43 +08:00
|
|
|
|
|
|
|
|
|
2026-08-03 03:50:20 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-05-25 17:46:17 +08:00
|
|
|
|
2009-03-31 14:33:13 +11:00
|
|
|
|
2019-09-16 13:15:14 -04:00
|
|
|
|
2025-08-29 16:04:16 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2016-11-18 16:16:11 +11:00
|
|
|
|
2014-09-30 14:23:59 +10:00
|
|
|
|
2022-07-14 11:06:57 -07:00
|
|
|
|
2012-10-11 13:34:00 +11:00
|
|
|
|
2021-10-04 23:34:53 +08:00
|
|
|
|
2017-05-08 11:56:55 +02:00
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-25 10:46:15 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
2009-12-14 12:49:55 +11:00
|
|
|
|
2009-04-14 12:01:53 +10:00
|
|
|
|
2023-08-25 11:09:50 +08:00
|
|
|
|
|
|
|
|
|
2024-06-26 16:26:22 +02:00
|
|
|
|
2022-07-23 08:24:29 +02:00
|
|
|
|
|
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
2017-11-19 22:17:01 -08:00
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
|
|
|
|
|
2011-10-11 16:45:26 +11:00
|
|
|
|
2012-05-22 13:54:30 +10:00
|
|
|
|
2010-06-01 19:37:27 +10:00
|
|
|
|
dm-raid456, md/raid456: fix a deadlock for dm-raid456 while io concurrent with reshape
For raid456, if reshape is still in progress, then IO across reshape
position will wait for reshape to make progress. However, for dm-raid,
in following cases reshape will never make progress hence IO will hang:
1) the array is read-only;
2) MD_RECOVERY_WAIT is set;
3) MD_RECOVERY_FROZEN is set;
After commit c467e97f079f ("md/raid6: use valid sector values to determine
if an I/O should wait on the reshape") fix the problem that IO across
reshape position doesn't wait for reshape, the dm-raid test
shell/lvconvert-raid-reshape.sh start to hang:
[root@fedora ~]# cat /proc/979/stack
[<0>] wait_woken+0x7d/0x90
[<0>] raid5_make_request+0x929/0x1d70 [raid456]
[<0>] md_handle_request+0xc2/0x3b0 [md_mod]
[<0>] raid_map+0x2c/0x50 [dm_raid]
[<0>] __map_bio+0x251/0x380 [dm_mod]
[<0>] dm_submit_bio+0x1f0/0x760 [dm_mod]
[<0>] __submit_bio+0xc2/0x1c0
[<0>] submit_bio_noacct_nocheck+0x17f/0x450
[<0>] submit_bio_noacct+0x2bc/0x780
[<0>] submit_bio+0x70/0xc0
[<0>] mpage_readahead+0x169/0x1f0
[<0>] blkdev_readahead+0x18/0x30
[<0>] read_pages+0x7c/0x3b0
[<0>] page_cache_ra_unbounded+0x1ab/0x280
[<0>] force_page_cache_ra+0x9e/0x130
[<0>] page_cache_sync_ra+0x3b/0x110
[<0>] filemap_get_pages+0x143/0xa30
[<0>] filemap_read+0xdc/0x4b0
[<0>] blkdev_read_iter+0x75/0x200
[<0>] vfs_read+0x272/0x460
[<0>] ksys_read+0x7a/0x170
[<0>] __x64_sys_read+0x1c/0x30
[<0>] do_syscall_64+0xc6/0x230
[<0>] entry_SYSCALL_64_after_hwframe+0x6c/0x74
This is because reshape can't make progress.
For md/raid, the problem doesn't exist because register new sync_thread
doesn't rely on the IO to be done any more:
1) If array is read-only, it can switch to read-write by ioctl/sysfs;
2) md/raid never set MD_RECOVERY_WAIT;
3) If MD_RECOVERY_FROZEN is set, mddev_suspend() doesn't hold
'reconfig_mutex', hence it can be cleared and reshape can continue by
sysfs api 'sync_action'.
However, I'm not sure yet how to avoid the problem in dm-raid yet. This
patch on the one hand make sure raid_message() can't change
sync_thread() through raid_message() after presuspend(), on the other
hand detect the above 3 cases before wait for IO do be done in
dm_suspend(), and let dm-raid requeue those IO.
Cc: stable@vger.kernel.org # v6.7+
Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Signed-off-by: Xiao Ni <xni@redhat.com>
Acked-by: Mike Snitzer <snitzer@kernel.org>
Signed-off-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/r/20240305072306.2562024-9-yukuai1@huaweicloud.com
2024-03-05 15:23:05 +08:00
|
|
|
|
2023-10-10 23:19:58 +08:00
|
|
|
|
2011-10-11 16:47:53 +11:00
|
|
|
|
2024-03-05 15:22:59 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2012-07-31 09:08:14 +02:00
|
|
|
|
2014-10-29 18:51:31 -05:00
|
|
|
|
2023-10-10 23:19:53 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2015-04-14 10:43:55 -05:00
|
|
|
|
2017-12-27 10:31:40 +01:00
|
|
|
|
2012-07-26 11:12:18 +02:00
|
|
|
|
2023-03-06 14:03:17 +01:00
|
|
|
|
md raid0/linear: Mark array as 'broken' and fail BIOs if a member is gone
Currently md raid0/linear are not provided with any mechanism to validate
if an array member got removed or failed. The driver keeps sending BIOs
regardless of the state of array members, and kernel shows state 'clean'
in the 'array_state' sysfs attribute. This leads to the following
situation: if a raid0/linear array member is removed and the array is
mounted, some user writing to this array won't realize that errors are
happening unless they check dmesg or perform one fsync per written file.
Despite udev signaling the member device is gone, 'mdadm' cannot issue the
STOP_ARRAY ioctl successfully, given the array is mounted.
In other words, no -EIO is returned and writes (except direct ones) appear
normal. Meaning the user might think the wrote data is correctly stored in
the array, but instead garbage was written given that raid0 does stripping
(and so, it requires all its members to be working in order to not corrupt
data). For md/linear, writes to the available members will work fine, but
if the writes go to the missing member(s), it'll cause a file corruption
situation, whereas the portion of the writes to the missing devices aren't
written effectively.
This patch changes this behavior: we check if the block device's gendisk
is UP when submitting the BIO to the array member, and if it isn't, we flag
the md device as MD_BROKEN and fail subsequent I/Os to that device; a read
request to the array requiring data from a valid member is still completed.
While flagging the device as MD_BROKEN, we also show a rate-limited warning
in the kernel log.
A new array state 'broken' was added too: it mimics the state 'clean' in
every aspect, being useful only to distinguish if the array has some member
missing. We rely on the MD_BROKEN flag to put the array in the 'broken'
state. This state cannot be written in 'array_state' as it just shows
one or more members of the array are missing but acts like 'clean', it
wouldn't make sense to write it.
With this patch, the filesystem reacts much faster to the event of missing
array member: after some I/O errors, ext4 for instance aborts the journal
and prevents corruption. Without this change, we're able to keep writing
in the disk and after a machine reboot, e2fsck shows some severe fs errors
that demand fixing. This patch was tested in ext4 and xfs filesystems, and
requires a 'mdadm' counterpart to handle the 'broken' state.
Cc: Song Liu <songliubraving@fb.com>
Reviewed-by: NeilBrown <neilb@suse.de>
Signed-off-by: Guilherme G. Piccoli <gpiccoli@canonical.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
2019-09-03 16:49:00 -03:00
|
|
|
|
2023-03-06 14:03:17 +01:00
|
|
|
|
md raid0/linear: Mark array as 'broken' and fail BIOs if a member is gone
Currently md raid0/linear are not provided with any mechanism to validate
if an array member got removed or failed. The driver keeps sending BIOs
regardless of the state of array members, and kernel shows state 'clean'
in the 'array_state' sysfs attribute. This leads to the following
situation: if a raid0/linear array member is removed and the array is
mounted, some user writing to this array won't realize that errors are
happening unless they check dmesg or perform one fsync per written file.
Despite udev signaling the member device is gone, 'mdadm' cannot issue the
STOP_ARRAY ioctl successfully, given the array is mounted.
In other words, no -EIO is returned and writes (except direct ones) appear
normal. Meaning the user might think the wrote data is correctly stored in
the array, but instead garbage was written given that raid0 does stripping
(and so, it requires all its members to be working in order to not corrupt
data). For md/linear, writes to the available members will work fine, but
if the writes go to the missing member(s), it'll cause a file corruption
situation, whereas the portion of the writes to the missing devices aren't
written effectively.
This patch changes this behavior: we check if the block device's gendisk
is UP when submitting the BIO to the array member, and if it isn't, we flag
the md device as MD_BROKEN and fail subsequent I/Os to that device; a read
request to the array requiring data from a valid member is still completed.
While flagging the device as MD_BROKEN, we also show a rate-limited warning
in the kernel log.
A new array state 'broken' was added too: it mimics the state 'clean' in
every aspect, being useful only to distinguish if the array has some member
missing. We rely on the MD_BROKEN flag to put the array in the 'broken'
state. This state cannot be written in 'array_state' as it just shows
one or more members of the array are missing but acts like 'clean', it
wouldn't make sense to write it.
With this patch, the filesystem reacts much faster to the event of missing
array member: after some I/O errors, ext4 for instance aborts the journal
and prevents corruption. Without this change, we're able to keep writing
in the disk and after a machine reboot, e2fsck shows some severe fs errors
that demand fixing. This patch was tested in ext4 and xfs filesystems, and
requires a 'mdadm' counterpart to handle the 'broken' state.
Cc: Song Liu <songliubraving@fb.com>
Reviewed-by: NeilBrown <neilb@suse.de>
Signed-off-by: Guilherme G. Piccoli <gpiccoli@canonical.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
2019-09-03 16:49:00 -03:00
|
|
|
|
|
|
|
|
|
2012-07-26 11:12:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2014-03-29 10:20:02 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-01-04 16:10:19 -08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2017-02-13 16:21:49 -08:00
|
|
|
|
2017-04-05 19:21:03 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-01-24 11:02:34 +01:00
|
|
|
|
2024-03-03 07:01:49 -07:00
|
|
|
|
2017-04-05 19:21:03 +02:00
|
|
|
|
2020-06-06 15:00:24 +02:00
|
|
|
|
2023-10-10 23:19:44 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-10-10 23:19:58 +08:00
|
|
|
|
2023-10-10 23:19:44 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-10-10 23:19:58 +08:00
|
|
|
|
2023-10-10 23:19:44 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-10-10 23:19:58 +08:00
|
|
|
|
2026-04-15 16:03:19 +02:00
|
|
|
|
2023-10-10 23:19:44 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2023-10-10 23:19:58 +08:00
|
|
|
|
2023-10-10 23:19:44 +08:00
|
|
|
|
|
|
|
|
|
2020-06-07 17:31:19 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2020-06-07 16:33:01 +02:00
|
|
|
|
2020-06-06 15:00:24 +02:00
|
|
|
|
2020-06-07 17:31:19 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-06-13 10:48:22 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-03-03 07:01:43 -07:00
|
|
|
|
|
|
|
|
|
2020-06-07 17:31:19 +02:00
|
|
|
|
|
|
|
|
|
2020-06-06 15:00:24 +02:00
|
|
|
|
2026-05-01 13:46:49 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-03-03 07:01:42 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-08-29 16:04:17 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-03-03 07:01:40 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-03-03 07:01:42 -07:00
|
|
|
|
2024-03-03 07:01:40 -07:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-10-31 11:31:08 +08:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-03-03 07:01:41 -07:00
|
|
|
|
|
|
|
|
|
2024-03-03 07:01:42 -07:00
|
|
|
|
2024-03-03 07:01:49 -07:00
|
|
|
|
2024-03-03 07:01:41 -07:00
|
|
|
|
|
|
|
|
|
2009-04-14 12:01:53 +10:00
|
|
|
|
