Currently I have to configure a Sun Enterprise T5220 as our new “developemnt environment”, replacing our V440 (Oracle) and the T1000 (Weblogic). We chose a Sun CMT machine as we wanted to stay close to production in regards of architecture (processor type/OS etc).
The machine will have to run an Oracle instance, and some containers for the Weblogic environments (we have an integration env, daily build env + some development environments for projects). The new T5220 ist packed with a 1.4GHz T2 Niagara, 64GB of memory, 2x146GB for the OSes, and 6x300GB for the database and NFS.
Given that, the ideal setup sounds like:
- 2 Guest LDOMs, where:
- 1 Oracle-LDOM
- 1 Weblogic LDOM, using Solaris zones to seperate the environments
- The setup is 100% ZFS
- The control domain runs on one slice of the 146GB drives in ZFS raid1
- The guest domain roots run on the other slice of that disk, seperate zpool, exported as ZVOL
- The 6 300GB disks are exported as raw disk slices (EFI labeled), and formed into a ZFS raidz inside the Oracle LDOM
- Inside that raidz zpool, there is a ZFS with recordsize=8k for the Oracle datafiles, and a ZFS with 128k blocksize for the redo logs
Regarding the database, this is pretty similar to our current setup, a SunFire V440 with a ST2540 FC-Storage attached, running ZFS on top of the hardware RAID5. Similar in regards of the filesystem, at least. ZFS runs very well on the V440.
Okay, now, after setting this up, it turned out that the DB performance is unacceptable. Absolutely horrible to be honest. We’re doing large imports of our production database on a regular basis, and on the V440 it takes about 50 minutes, now we’re up to 150 minutes on the T5220.
Here are some numbers. Simple dd testing: creating a 15GB file.
V440 with St2540 RAID5 volume, exported via 2x2Gbit FC, configured as a simple ZFS
$ time dd if=/dev/zero of=ddtest.file bs=8k count=2000000 2000000+0 records in 2000000+0 records out real 2m32.380s user 0m3.021s sys 1m27.533s $ echo "((16384000000/152)/1024)/1024"|bc -l 102.79605263157894736842
T5220 with 10k/rpm local SAS, exported as raw disk slice into guest LDOM and configured as ZFS RAIDz:
$ echo "((16384000000/336)/1024)/1024"|bc -l 46.50297619047619047619
Now things get strange. Inside the control domain, onto the ldompool:
$ echo "((1638400000/35)/1024)/1024"|bc -l 44.64285714285714285714
That is worse than I get with a Linux guest in VirtualBox onto a virtual drive. Not acceptable for 300€ 10k rpm/s SAS drives.
Me asking for help and further infos at Sun forums
Anyways, I’m going to attach the documented step-by-step guide on how to setup all this from scratch (M$-Word .doc, in German, sorry)