In the process of upgrading from existing physical servers to an (almost) fully virtualized environment, a new backup system was required.
The existing backup system is USB, utilizing 1TB Tandberg RDX QuikStor cartridges managed by Backup Exec on one of the physical servers running Server 2008 R2.
Backups are run nightly, with one cartridge weekly taken off site.
There were two physical servers, Microsoft Exchange, and the Active Directory Domain Controller / Fileserver.
The new system has one physical Server 2016 domain controller, two VMWare ESXi hosts, an HP SAN, a virtual Server 2016 domain controller, the virtual Microsoft Exchange server, and a virtual MX integration box that emails voicemail from the VOIP system (was moved p2v, previously not backed up).
The goal is to produce scheduled daily backups in the rack on storage separate from the SAN, then replicate the backup offsite to another datacenter that houses the public side servers over a 100Mbit fiber WAN link.
Unitrends backup solution seemed ideal for this, I had used the free version for testing and found that the backup fit within the 1TB storage limitation quite easily.
The FreeNAS box is an HP server with 20GB of RAM, two 8-core Intel Xeon processors, a 4-port 1Gbit Broadcom card, the integrated HP P410i SAS RAID controller, and 4 600GB 10k SAS drives.
The HP P410i does not support HBA mode or passthrough, so we must configure the RAID volumes using the HP Array Configuration Utility (provided on the bootable HP Smart Start ISO) rather than creating software RAID with FreeNAS.
The configuration for testing was RAID 1+0, which produced a 1.2TB volume with very good write and read speeds – while this was fine for the free version, I did need the extra capacity that only RAID 5 could provide. Initially I encountered very slow write performance, a paltry 10MB per second. This was suspiciously close to the maximum throughput for a 100Mbit network link, so I suspected that something was wrong with my network configuration.
My initial configuration for the FreeNAS box was using the Link Aggregation Protocol to group three 1Gbit ports on the box and switch, however as I discovered this is not supported for iSCSI. Link aggregation was only providing the bandwidth of a single 1Gbit link, but unfortunately was not the cause for the slow writes.
I discovered that the controller cache was incorrectly allocated only to reads by default in the ACU, which amplified the already terrible write penalty for RAID 5 in spite of having the drive cache and every other option enabled.
The solution to make the best use of the multiple network links is iSCSI MPIO, for VMWare ESXi and FreeNAS this can be accomplished by configuring each port on the FreeNAS box with it’s own subnet (example 172.16.1.x, 172.16.2.x, 172.16.3.x, all /24) and assigning them each to the iSCSI portal.
On the VMWare side, a matching VMKernel adapter for each subnet on each host is required, bound to their own physical Ethernet ports. Once complete, bind the new network ports to the iSCSI adapter, add the target IP addresses for the portal and rescan, then set the multipathing policy for the device to Round Robin.
All of these network and cache configuration changes were made with the Unitrends VM shut down without negatively affecting the Unitrends Backup storage volume. Once everything was powered back up a quick 2GB test backup succeeded in writing 30MB / second using approximately 150Mbit /second each on two of the 1Gbit Ethernet ports. A rather dramatic improvement.
zcomputerwiz 