HP Z820 Upgrades

Since I bought myself a HP Z820 Workstation I've made several upgrades to it, in fact - most of the internals are new. The first round of upgrades I did were the following:

• Upgraded the motherboard from a REV 1.00 to a 1.03. The new 1.03 board has a more recent bootblock and supports E5v2 Xeons, Intel AMT 8.x and 1886MHz RAM.
• Upgraded both CPU's from two Xeon E5-2690's to Xeon E5-2680v2's, improving not only core-count (4 extra cores + 8 more threads) but overall performance. The machine now has 20 cores and 40 threads. I didn't go with the 12-core E5-2697v2 option because the raw core speed (base clock and turbo boost) would be too low and the 10-core Xeon E5-2690v2 were almost double the price of the E5-2680v2's for next-to-zero gain - except in some heavily multithreaded workloads (12+ threads). The E5-2680v2's also has a surprisingly low 115W TDP which is a really nice bonus. A benefit from keeping the TDP down is that air-cooling is guaranteed to suffice so watercooling isn't necessary (required for the E5-2687Wv2's according to the specification). This will avoid certain risks a couple years down the line since HP's watercooling is more likely than air cooling to fail over time and will be much more difficult to find replacements parts for as time goes on and stocks dwindle. Another smart option would've been the E5-2667v2's but i favored the lower TDP and higher core count over the improved core speed.
• Replaced the fan shroud with a new and unused one, lowering noise output slightly.
• Upgraded RAM from 32GB (4x8GB) 1600Mhz Hynix with 64GB (8x8GB) 1866MHz Micron RAM.
• Upgraded the boot drive from a HP 500GB SAS drive to a 512GB Samsung 950 Pro M.2 NVMe (that has the required NVMe OOPROM). This, along with a Aqua Computer Kryo M.2 PCIe adapter made a great combo. This should even outperform a HP Turbo Drive G1 that has the slower Samsung SM951. The HP Performance Advisor even detects the 950 Pro as an HP Turbo Drive for some reason.
• Upgraded the Quadro 4000 with a GTX 1080 Founders Edition. Not a workstation card so it's isn't ideal for CAD work but useful for pushing high resolutions on 3 1440p displays.
• Got myself a rather hard to get dual 6-pin to single 8-pin adapter that enables use of cards that has an 8-pin power connector such as the GTX 1080 (Update: Avoid these, system becomes unstable due to insufficient power delivery - go for a dual 6-pin to 8-pin instead).
• Replaced both sidepanels with the more rare black ones, giving the workstation a stealthy look. This is one of the most aesthetically pleasing upgrades you can make - highly recommended these parts if you can find them.
• Bought the slot-loading optical drive carrier option to replace the included, and rather dull, 5.25" inch DVD burner with a slot-loading Blu-ray burner.
• Bought the Z840 Intel vPro Wireless card option - not only giving it the ability connect wirelessly but also bluetooth (I still use ethernet cables though for performance). Not sure if the Z820's AMT supports this card but I don't use AMT anyway.

In my second round of upgrades I got:

• Upgraded the GTX 1080 Founders Edition with a Titan Xp. Titans are rediculous cards, I know, but they're also a surprisingly good fit for Z820 while being a lot cheaper than a Quadro P5000/P6000 - and if you get a second hand one for a good price? Why not. The Xp even has a slight performance edge over the P6000 in a number of cases. This is about as high as I think you should go with the Z820 in terms of graphics given the limited power delivery options you have (3x6-pin power connectors) all of which are put to use with the Titan XP. The latest high-end RTX cards require dual 8-pin power delivery which is too much for the Z820. I suppose you might be able to use a dual molex to single 6-pin adapter daisy chained with yet another dual 6-pin to 8-pin adapter and I'm still not sure the two molexes would give you enough power (you need 75w) to make it stable - the custom PSU in these machines might not follow the standards. In theory, it should work, but I wouldn't count on it and the cost of an RTX 2080 Ti card is just way too much when the Titan Xp is a perfect fit. The great thing about the Titan Xp in a Z820, however, is that the Z820's cooling augments the Xp's blower design and keeps it quiet and cool so the thermal throttling these particular cards are known for becomes a non-issue. The best conceivable card for this machine would of course be the P6000 but the price tag is just so over the top for a mere enthusiast. The RTX 2080 Ti on the other hand would probably be a better choice for a Z840 if you could track down a blower version. I suspect the Z820 would bottleneck a RTX 2080 Ti, keeping the cards from living up to it's full potential - I'm already seeing a bit of this with the Titan Xp.
• Replaced the slot-loading Blu-ray drive with the original slot-loading dvd drive the carrier was supposed to come with. Turns out the blu-ray drive I bought was a rather shitty one, oh well.
• Added a second Samsung 950 Pro M.2 drive. Also mounted with a Aqua Computer Kryo M.2 PCIe adapter.
• Added a Samsung 860 Pro 500GB SATA drive, will probably be removed later due to already having two M.2's.
• Added a 10TB HGST SAS (Helio 10) drive.

In my third round of upgrades I bought:

• Dual Xeon E5-2667v2's, I sacrificed four cores and eight threads for higher base/turbo clocks. Performance is about equal to what you would expect from a pair of Xeon E5-2687W v2's eventhough the base clock is 100MHz lower. On UserBenchmark my personal highest CPU score was 97 and the highest anyone has achieved is a score of 100 with a pair of E5-2687W v2's and those have a higher TDP and are limited to 256GB RAM unlike the E5-2667 v2's.
• Another 8x8GB 64GB Micron RAM Kit, topping out at 128GB total.
• Dual HP Z Coolers, they really are better and gave me equal temperatures on the E5-2667v2's to the stock coolers with the lower TDP Xeon E5-2680v2's. My personal opinion on these though? Don't waste your money unless you absolutely need a couple of degrees of improvment. Way too expensive for what they are.

Accessories I got are

• 3x HP Z24nq (Quad HD) IPS displays. These are 60Hz displays and might be suboptimal for gaming, but excellent for professional work.
• A triple monitor stand.
• Creative Katana Soundbar. Amazing sound for what it is and does not take up much space with minimal cabling - I recommend this one.
• Corsair RGB Mechanical keyboard.
• Corsair RGB Mouse.
• Corsair RGB headphone stand.
• Corsair M300 extended mouse pad.
• Corsair HS50 headphones, as daily drivers.
• Audio Technica ATH-M50x headphones, for professional work.

Some regrets

I regret buying a cheap triple monitor stand off eBay instead of looking around more and checking reviews. Quite frankly it was utter garbage and poorly designed. I had to get a new one and this time I followed other people's recommendations.

I regret buying so many "gamer" accessories. I don't even game all that often so they look a bit out of place hooked up to a professional workstation. Corsairs build quality is top notch and some of the best accessories I've ever used but the gamer features are silly and in some cases poorly thought out. The RGB features are proprietary on both the Creative Katana and Corsair gear so multiple softwares are required to configure lighting. The rediculously fast polling rate on the keyboard forces you to leave it in BIOS mode at all times because otherwise keystokes don't register during the early boot process, preventing you from entering BIOS, but BIOS mode also cannot be enabled when you use the iCUE software so flipping back and forth between modes still becomes a neccessary thing. The headset stand is unable to save RGB configuration on the device itself so during boot you get the default rainbow effect and the iCUE software is thus required to run at all times to sustain the RGB effect you configured (Why on the headset stand only?! Annoying as hell). Also, the headset stand apparently presents itself as an input device over USB so if it's connected and powered on during boot it might, annoyingly, cause conflict with your actual keyboard and once again make your system unable to register keystrokes. Still, I can't go back to generic OEM keyboards because the build quality of both the mouse and keyboard are just that good and the typing experience is second to none. Eventhough the headset stand is so annoying I can't complain about the excellent build quality, aesthetics and the awesome built-in DAC. This leaves me torn and longing for accessories with the same build quality yet with all the gamer features stripped out and paired with a stronger focus on compatibility with a wider range of systems.

HP Z820 Workstation & GTX 1080 - Will it work?

The short answer is: Not out of the box, but it will work with a very specific adapter.

The long answer is: To power a GTX 1080 in a Z820 Workstation a single 6pin-to-8pin PCIe power adapter will not be sufficient. There has still been some discussion about if it might work because there isn't all that much information about the PSU and how it's configured internally. Sometimes a 6-to-8pin adapter can work with certain PSU's that exceed the normal specifications of the 6pin connector - perhaps to enable use of such adapters. On the Z820 and the 1125w PSU it doesn't work.

The system boots fine and is usable until you stress the GPU at which point it will promptly bluescreen. To be fair, on paper it should not work because a 6pin connector provides only 75w and the PCIe bus gives you another 75w - amounting to 150w. The GTX 1080, on the other hand, needs about 170-180w under full load. So, how does one even use a >150w card on the Z820? You will need something like this

A dual 6pin-to-8pin adapter, but unfortunately they are about as easy to find as pink unicorns (try eBay?). These provides you with 225w of power (75+75+75w) and will make your card run stable with 35-45w overhead. Nvidia does not recommend these adapters due to "grounding requirements" of these power hungry cards - as such you will not find these being sold by Nvidia nor will you get one with your GTX 1080 purchase. These adapters do work, so if you're out of options and are willing to risk a configuration that might not be covered under warranty then a dual 6pin-to-8pin adapter is pretty much your only option. These adapters are probably difficult to find because most people would buy a new PSU to fulfill the GTX 1080's requirements. As a Z820 owner, however, you don't have that luxury because of the non-standard PSU formfactor.

CentOS 7 & GTX 1080

CentOS 7 does not currently support Nvidia Pascal GPU's out of the box so if you are running into issues with the graphical installer then it's a safe bet that it's because you have a Pascal card in your machine. You can work around this by using the text-based installer but partitioning isn't as full-featured so you might be forced to use the automatic partitioning option.

Once CentOS is installed you can simply download the proprietary Nvidia drivers - which work quite nicely I might add - and be up and running with your desktop in no time. As long as you know how to navigate the commandline until you get the proper drivers installed then it shouldn't be much of a challange.

I have a GTX 1080 in my workstation and the graphical installer hung while systemd was initializing. You'd think at least VESA graphics would work on Pascal GPU's but it doesn't - at least not currently. Hopefully the CentOS installation disks will be updated at some point to at least include rudimentary GTX 10XX support.

List of maker boards

Ever since the Raspberry Pi became popular several new ARM development boards have hit the market, some even predate the Pi but gained more popularity as the Pi paved the way into mainstream. It's getting to the point where it's difficult to keep track of all the new offerings so this post is a reference list for myself which i plan on updating as new interesting devices become available. It only lists hacker friendly mainboards and not ARM devices in general. It would be fun to add some MIPS based boards too - some do exist, but the ARM based devices tend to be more interesting and popular. I list only the models with the highest spec available.

With the performance boost of the Raspberry Pi 2 & 3 most alternative maker boards loose their appeal - but the various outputs or the presence of gigabit ethernet still makes some alternatives into better options. Recently Udoo has caught my attention as a project with interesting ideas and their recent x86 model has great potential for real-world deployment where the Raspberry is too limited - digital signage springs to mind. I've known about Udoo for years but now that their x86 board is on it's way I think it has excellent potential even though it's a lot more expensive than the Pi.

Board	URL	CPU	Cores	GPU	RAM
Raspberry Pi 3	Website	ARM Cortex A53	4@1.2GHz	Broadcom VideoCore IV	1GB
Beagleboard	Website	ARM Cortex A8	1@1GHz	PowerVR SGX530	512MB
Pandaboard	Website	ARM Cortex A9 MPCore	2@1.2GHz	PowerVR SGX540	1GB
ArndaleBoard	Website	ARM Cortex A15	2@1.7GHz	Mali-T604	2GB
Wandboard	Website	Freescale i.MX6 Quad	4@1GHz	Vivante GC 2000	2GB
Udoo	Website	Freescale i.MX6 Quad	4@1GHz	Vivante GC 2000	1GB
HummingBoard	Website	Freescale i.MX6 Quad	4@1GHz	Vivante GC 2000	2GB
pcDuino3	Website	AllWinner A20 (ARM Cortex A7 Dual Core)	2@1GHz	Mali-400MP2 (Dual Core)	1GB
Banana Pi	Website	AllWinner A20 (ARM Cortex A7 Dual Core)	2@1GHz	Mali-400MP2 (Dual Core)	1GB
Cubieboard	Website	AllWinner A20 (ARM Cortex-A7 Dual Core)	2@1GHz	Mali-400MP2 (Dual Core)	2GB
OLinuXino	Website	??	??	??	??
APC Rock	Website	??	??	??	??
Origenboard	Website	??	??	??	??
ODROID	Website	??	??	??	??
CuBox-i	Website	??	??	??	??

I'll keep updating this.

How to force a downgrade of the BIOS on a HP Z Workstation

Important note: I've only done this on a HP Z820 so I cannot guarantee that it's possible on other models.

I ran into a seemingly rare problem when i upgraded to the latest BIOS on my Z820 Workstation - Intel AMT stopped working. I didn't notice this at first so I could only assume the BIOS upgrade screwed things up. My machine originally had a really old original 1.x BIOS on it and I only did one intermediate (and mandatory) BIOS update before I jumped to the very latest version and I think this huge leap between versions might have contributed to the breakage - that along with the fact that I hadn't flashed the latest Intel AMT firmware along with the newer BIOS. To my dismay the HP download page said that BIOS downgrades were no longer allowed and sure enough - you really can't downgrade - not even when flashing from inside the BIOS. It was sloppy of me not to check this beforehand.

So what was I to do? I tried everything I could think of and finally discovered how to get around this and in the end it wasn't even all that difficult, obvious even. You see, HP Z Workstations allows you to bridge some pins on the motherboard to put the machine in emergency recovery mode (look it up in the manual) to rescue the computer from a bad BIOS flash. In emergency recovery mode you bypass the BIOS entirely and use the bootblock to load an old BIOS image from a USB stick - one that doesn't prevent downgrades which then, in turn, allows you to flash any BIOS version. The emergency recovery procedure is well documented. It probably didn't occur to the HP engineers that the bootblock should be able to prevent older BIOS's from loading because that in itself would involve risks if a new and badly flashed BIOS along with the bootblock prevented you from going to an earlier (working) version.

The procedure is as follows:

1. Bridge the emergency recovery pins on motherboard (check manual for instructions).
2. Prepare a USB stick with an older BIOS that doesn't prevent downgrades - this is very important. Only the last couple of versions prevent downgrades so you won't have to go too far back. Changelogs will note if BIOS no longer allows downgrade.
3. Boot your workstation with USB stick connected to a USB port, the emergency recovery mode will now start the computer seemingly like normal - except the older BIOS is now running off your USB stick and not the one flashed to your motherboard. Pay attention during boot to see which BIOS version is loaded as you boot.
4. Download and flash any BIOS version you like once booted into your operating system, effectively downgrading the BIOS.
5. Unbridge pins on motherboard.

As far as I know the BIOS still needs to be a signed, original HP BIOS and I assume the bootblock verifies this. Still, I haven't tried any non-official firmware so I cannot say for sure.

Anyway, Intel AMT started working again with the latest BIOS after I made smaller, incremental, updates between versions and somewhere in between I managed to flash the latest Intel AMT firmware so this little trick solved my problem. I can't remember all the BIOS versions I jumped between to get AMT up and running again but it shouldn't be difficult to figure out - just keep downgrading to older BIOS's until you find one that makes it work again and then do upgrades a bit more carefully - and don't forget to check the device manager to make sure the Intel Management Engine is still working. Regardless of what HP wants you to think you will not break the machine by going from the very latest BIOS to a much older one - older BIOS's worked just fine for me. There are also other reasons you might want to downgrade the BIOS - some applications are certified to run on very specific BIOS versions so this way you can downgrade a new BIOS to one that is certified for an application that requires it. Or maybe a newer version was just unstable? That's something I've experienced on occasion.

HP insist that "you cannot downgrade" but considering that others might run into similar problems I figured this trick could be important to share. What's interesting is that there is no way for HP to actually prevent you from doing this in the future unless they're willing to also update the bootblock and they are notoriously resistant to do this - and for good reason, because it opens up the possibility of permanently bricking a machine in the event of a bad flash of said bootblock. If, however, they did release bootblock updates with version checking many would at the same time rejoice as newer bootblocks would finally allow early revision motherboards to support v2 Xeons, which they technically can do but are not allowed to by the older 2011 bootblock. Still, my guess is that HP will never allow bootblock updates and this also means is that they can never stop you from downgrading the BIOS regardless of what they claim.

HP Z820 Workstation Fan Inventory

I've been considering a complete fan replacement on my HP Z820 Workstation but there are some issues to be aware of. First of all there is a whopping 12 fans in total and HP uses a non-standard OEM pinout, which makes the modifier numbers especially important. The second issue is that different workstation revisions use different fans - some fan shrouds seem to be equipped with Delta fans only while others are a mix of Delta, Nidec and AVC. Surprisingly the model numbers used on individual fans is poorly documented, presumably because owners are expected to buy an entire fan shroud replacement if one or more fans go bad or a kit of front or back facing fans. If the fans go bad on the power supply you are probably meant to get a new one. When you have an expired warranty, options become a bit more limited so I find it far more compelling and cost effective to just buy the individual fans and replace them myself, and to do that you'll need to know exactly what models that work.

My goal here is to - first and foremost - identify all the fans, including their positions, in my own workstation and then all alternatives that can work as drop-in replacements. The HP Z820 seems to be able to use the exact same fans as the newer Z840 and vice versa. A number of these fans should work in the Z600/Z400-series and the older Z800. I cannot confirm this, however, as I don't have a Z800 so my main focus will be on the Z820.

Why bother with this? Delta fans, for example, have a good reputation but are unfortunately known to be noisy so owners with those might be interested in the quieter Nidec options - especially if you do audio work and need a quieter system. Those of us with Nidec fans might be interested in replacing them with Delta's for other reasons - perhaps the better longevity, price and availability are deciding factors. I wanted to document the cooling fans to make things easier for people who want to do the replacement themselves so having the exact model information should keep your workstation within HP's specifications (CFM, RPM etc.). The fan numbering scheme i use in the images is arbitrary. I think information about alternative fans to those i happen to have is somewhat complete but I have no way of knowing if I've identified all variants HP has used in these workstations. I don't have all modifier numbers yet but I will update this post when/if I do.

Update: I went with the option of replacing the entire shroud because the combined shipping costs and rarity of some fans was making things difficult. I got a great deal on eBay on a brand new shroud for the dual cpu option.

The Shroud

The Case

The Powersupply

FAN1, FAN2 & FAN4

  Dimensions: 60mm (Blower-style) 4-pin
  Model: Delta BUB0712HF
  Modifier number: -BE04
  HP P/N: 670051-001 REV 1

FAN3

Note: The AVC fan in my shroud doesn't seem to have a HP product number which suggests that the generic model might have the same pinout. I did manage to find the PN from other sources, however.

  Dimensions: 60*60*25mm 4-pin PWM
  Model: AVC DS06025B12U
  Modifier number: P063
  HP P/N: 670050-001 Rev.A

  Alternative Delta Model
  Dimensions: 60*60*25mm 4-pin PWM
  Model: Delta AFB0612EH
  Modifier number: ??
  HP P/N: ?? (PN Required for HP OEM model)

FAN5, FAN6, FAN9 & FAN10

Note: The Nidec fans are rated at ~38db while Delta's are around ~45-49db and both run at 3800 RPM. Some sources suggest the Nidec fans use sleeve bearings while others that it's a ball bearing - I'm pretty sure they're sleeve bearing. The 92mm Nidec's have a MTBF of 45.000h which is nearly equal to the 50.000h you are likely to get from the Delta. I could not find the specifications for these exact OEM fans so these are my best guesses based on the specifications of the non-OEM equivalents of the Nidec and Delta fans. In other words, I cannot guarantee the above statements about the bearing, MTBF and noise profile to be 100% correct.

  Dimensions: 92*92*25mm 4-pin
  Model: Nidec T92T12MS3A7-57A03
  Modifier number: 2223G
  HP P/N: 647113-001 REVA (or REVB)

  Alternative Delta Model
  Dimensions: 92*92*25mm 4-pin
  Model: Delta QUR0912VH
  Modifier number: -AK59
  HP P/N: 647113-001 REV 0A

  Alternative Nidec Model (New revision that should work)
  Dimensions: 92*92*25mm 4-pin
  Model: Nidec FAN A T92T12MS3A7-57A03
  Modifier number: ??
  HP P/N: 644315-001 REVB

  Alternative Delta Model (Used in Z800 that should work)
  Dimensions: 92*92*25mm 4-pin
  Model: Delta QUR0912VH
  Modifier number: -8C2T
  HP P/N: 468763-001

FAN7 & FAN8

Note: FAN7 & FAN8 come as a pair with wires going to the same proprietary 6-pin connector

  Dimensions: Dual 92*92*25mm fans to one 6-pin
  Model: Nidec FAN B T92T12MS3A7-57A03
  Modifier number: 2429H F4 / 4X30H G4
  HP P/N: 644315-001 REVB

  Alternative Delta Model
  Dimensions: Dual 92*92*25mm fans to one 6-pin
  Model: Delta QUR0912VH
  Modifier number: -BL3H
  HP P/N: 644315-001 REV 0B (or 0A)

FAN11 & FAN12

Note: The powersupply contains two identical Delta fans of the below model, these were found in the 1125W PSU and are likely also to be found in the less powerful 800W PSU. Given that Delta built these power supplies I seriously doubt any other fan manufacturers are used.

  Dimensions: 80*80*25mm 4-pin
  Model: Delta QUR0812SH
  Modifier number: -HE00
  HP P/N: None

Was udev a bad idea? Not at first.

I'm not an expert when it comes to device handling on Linux but I have written udev rules in the past (since 071 and earlier) and do have some familiarity, and at the time udev was first introduced it looked like an excellent solution. In fact, I felt it was a welcomed change from devfs. Not so long ago time came to upgrade to 105, and boy was I surprised - these days udev really looks like shit.

It sure doesn't seem like udev's design has scaled very well. For instance, udev rules you wrote even 6 month ago are unlikely to work with the most recent version because they keep changing the syntax every few releases. Why would stuff like $modalias suddenly be changed to a much uglier $env{MODALIAS}? I'm curious to know why anyone would favor such a syntax.

Also, udev 071 consisted of a fairly small set of binaries, but later releases grew to pretty much one helper program for each device class, and you are now expected to do stuff like copy pre-made symlinks and device nodes manually into the /dev directory. If anything, wasn't that supposed to be udev's job in the first place? A nice improvement is that udev 071+ completely replaces hotplug for the 2.6 kernel - which is perhaps the reason for all the helper programs.. I'm not entirely sure. I still think it could have been done a lot cleaner, though.

udev rules used to be something that could be written in a very clean and comprehensible style, but now you'll get a headache just by looking at basic examples. Quite frankly I'm starting to dislike it more and more and sometimes find myself wishing for it to just go away. After looking at FreeBSD and how clean /dev is kept I'm starting to wonder if it really was such a great idea to abandon devfs all together.

Update: The above is an old post. Since then udev was merged into systemd - a rat's nest of a init system that I'm forced to use daily, fortunately the eudev fork exists.