Friday, January 6, 2017

How to force a downgrade of the BIOS on a HP Z Workstation

Important note: I've only done this on a HP Z820 so I cannot guarantee that it's possible on other models.

I ran into a seemingly rare problem when i upgraded to the latest BIOS on my Z820 Workstation - Intel AMT stopped working. I didn't notice this at first so I could only assume the BIOS upgrade screwed things up. My machine originally had a really old original 1.x BIOS on it and I only did one intermediate (and mandatory) BIOS update before I jumped to the very latest version and I think this huge leap between versions might have contributed to the breakage - that along with the fact that I hadn't flashed the latest Intel AMT firmware along with the newer BIOS. To my dismay the HP download page said that BIOS downgrades were no longer allowed and sure enough - you really can't downgrade - not even when flashing from inside the BIOS. It was sloppy of me not to check this beforehand.

So what was I to do? I tried everything I could think of and finally discovered how to get around this and in the end it wasn't even all that difficult, obvious even. You see, HP Z Workstations allows you to bridge some pins on the motherboard to put the machine in emergency recovery mode (look it up in the manual) to rescue the computer from a bad BIOS flash. In emergency recovery mode you bypass the BIOS entirely and use the bootblock to load an old BIOS image from a USB stick - one that doesn't prevent downgrades which then, in turn, allows you to flash any BIOS version. The emergency recovery procedure is well documented. It probably didn't occur to the HP engineers that the bootblock should be able to prevent older BIOS's from loading because that in itself would involve risks if a new and badly flashed BIOS along with the bootblock prevented you from going to an earlier (working) version.

The procedure is as follows:
  1. Bridge the emergency recovery pins on motherboard (check manual for instructions).
  2. Prepare a USB stick with an older BIOS that doesn't prevent downgrades - this is very important. Only the last couple of versions prevent downgrades so you won't have to go too far back. Changelogs will note if BIOS no longer allows downgrade.
  3. Boot your workstation with USB stick connected to a USB port, the emergency recovery mode will now start the computer seemingly like normal - except the older BIOS is now running off your USB stick and not the one flashed to your motherboard. Pay attention during boot to see which BIOS version is loaded as you boot.
  4. Download and flash any BIOS version you like once booted into your operating system, effectively downgrading the BIOS.
  5. Unbridge pins on motherboard.

As far as I know the BIOS still needs to be a signed, original HP BIOS and I assume the bootblock verifies this. Still, I haven't tried any non-official firmware so I cannot say for sure.

Anyway, Intel AMT started working again with the latest BIOS after I made smaller, incremental, updates between versions and somewhere in between I managed to flash the latest Intel AMT firmware so this little trick solved my problem. I can't remember all the BIOS versions I jumped between to get AMT up and running again but it shouldn't be difficult to figure out - just keep downgrading to older BIOS's until you find one that makes it work again and then do upgrades a bit more carefully - and don't forget to check the device manager to make sure the Intel Management Engine is still working. Regardless of what HP wants you to think you will not break the machine by going from the very latest BIOS to a much older one - older BIOS's worked just fine for me. There are also other reasons you might want to downgrade the BIOS - some applications are certified to run on very specific BIOS versions so this way you can downgrade a new BIOS to one that is certified for an application that requires it. Or maybe a newer version was just unstable? That's something I've experienced on occasion.

HP insist that "you cannot downgrade" but considering that others might run into similar problems I figured this trick could be important to share. What's interesting is that there is no way for HP to actually prevent you from doing this in the future unless they're willing to also update the bootblock and they are notoriously resistant to do this - and for good reason, because it opens up the possibility of permanently bricking a machine in the event of a bad flash of said bootblock. If, however, they did release bootblock updates with version checking many would at the same time rejoice as newer bootblocks would finally allow early revision motherboards to support v2 Xeons, which they technically can do but are not allowed to by the older 2011 bootblock. Still, my guess is that HP will never allow bootblock updates and this also means is that they can never stop you from downgrading the BIOS regardless of what they claim.