Disk Speed and RAID
Virtual Devices and IOPS
As SoftNAS® is built atop of ZFS, IOPS (I/O per second) are mostly a factor of the number of virtual devices (vdevs) in a zpool. They are not a factor of the raw number of disks in the zpool. This is probably the single most important thing to realize and understand, and is commonly not. A vdev is a "virtual device". A Virtual Device is a single device/partition that act as a source for storage on which a pool can be created. For example, in VMware, each vdev can be a VMDK or raw disk device assigned to the SoftNAS® VM.
A multi-device or multi-partition vdev can be in one of the following shapes:
- Stripe (technically, each chunk of a stripe is its own vdev)
- A dynamic stripe of multiple mirror and/or RaidZ child vdevs
ZFS stripes writes across vdevs (not individual disks). A vdev is typically IOPS bound to the speed of the slowest disk within it. So if with one vdev of 100 disks, a zpool's raw IOPS potential is effectively only a single disk, not 100.
If the environment utilizes a hardware RAID which presents a unified datastore to VMware then the actual striping of writes occurs in the RAID controller card. Just be aware of where striping occurs and the implications on performance (especially for write throughput).
For information about RAID, see section RAID Considerations.
A common misunderstanding is that ZFS deduplication is free, which can enable space savings on a ZFS filesystems/zvols/zpools. In actuality, ZFS deduplication is performance on-the-fly as data is read and written. This can lead to a significant and sometimes unexpectedly high RAM requirement.
Every block of data in a deduplicated filesystem can end up having an entry in a database known as the DDT (DeDupe Table). DDT entries need RAM. It is not uncommon for DDTs to grow to sizes larger than available RAM on zpools that aren't even that large (couple of TBs). If the hits against the DDT aren't being serviced primarily from RAM (or fast SSD configured as L2ARC), performance quickly drops to abysmal levels. Because enabling/disabling deduplication within ZFS doesn't actually do anything to the data already committed on disk, it is recommended to not enable deduplication without a full understanding of its RAM and caching requirements. It may be difficult to get rid of later, after many terabytes of deduplicated data are already written to disk and suddenly the network needs more RAM and/or cache. Plan cache and RAM needs around how much total deduplicated data is expected.
Note: A general rule of thumb is to provide at least 2 GB of DDT per TB of deduplicated data (actual results will vary based on how much duplication of data is required).
Please note that the DDT tables require RAM beyond whatever is needed for caching of data, so be sure to take this into account (RAM is very affordable these days, so get more than may be needed to be on the safe side).
Extremely Large Destroy Operations - When destroying large filesystems, snapshots and cloned filesystems (e.g., in excess of a terabyte), the data is not immediately deleted - it is scheduled for background deletion processing. The deletion process touches many metadata blocks, and in a heavily deduplicated pool, must also look up and update the DDT to ensure the block reference counts are properly maintained. This results in a significant amount of additional I/O, which can impact the total IOPS available for production workloads.
For best results, schedule large destroy operations for after business hours or on weekends so that deletion processing IOPS will not impact the IOPS available for normal business day operations.