For reasons beyond me, I have a storage server set up with FreeNAS. I find it convenient to have all of my movies/music/TV shows on a PC that sees no action besides reading and writing to its drives. This weekend, while streaming a show from my storage server, the power went out. The server is connected directly to mains, so it went down – hard. To be fair, my Ubuntu based media center PC also died, never to recover. After a bit of troubleshooting and an annoying secondary problem, it seems to be up and stable.

My FreeNAS server is based on a 4 year old AMD 3200+ PC. Before setting everything up I made sure to clean the motherboard and CPU mount, applied new thermal grease on the CPU heatsink and upgraded the PSU to a Thermaltake 480W PSU – I really wanted to reduce chance of having any electrical/heat related issues that would take out the system. FreeNAS was loaded on a USB drive and a single 1.5TB drive was attached to the SATA bus. The server sat alone in a closet, connected via Gigabit ethernet to my home LAN and remained there. Headless, keyboard-less and monitor-less.

After the outage, the server did not respond to pings. I hooked it up to my monitor and attached a keyboard. It seemed like the BIOS did not even attempt to load an OS. I checked the BIOS settings and it seemed to revert to booting from HDD. Triumphantly I changed it to boot from USB and rebooted. Nothing. I then went ahead and restored BIOS defaults and again changed to boot from USB, just to start from a clean slate seeing as something freaky went on with the BIOS settings. No dice.

I tried yanking the USB key holding FreeNAS but that showed nothing. I then yanked the SATA cable from the HDD and finally something – the BIOS seemed to print something about Nvidia boot loader agent, but again hung. I knew something had shifted so I replugged the USB drive and voila – FreeNAS loaded. I let it load again with the HDD connected and it seemed I was successful.

Plot twist: FreeNAS reboots itself every 20 minutes

I patted myself on the back, called the wife, sat on the couch and loaded “The Wire” episode we were watching. After a few seconds, VLC crapped out indicating failure to read from file. The storage server was down again. I was smart and pessimistic enough not to disconnect the server from the monitor so I could get a glimpse of what was happening. After a bit of investigating, I saw that FreeNAS would reboot every ~15-20 minutes, claiming the following panic:

panic: ffs_blkfree: freeing free block

So it’s a file system issue, methinks. I then performed the following:

  1. Via the FreeNAS shell I ran “mount” to see which partition my storage mount was on (it was /dev/ad4p1)
  2. Via the FreeNAS web interface, I deleted the shares on the drive and deleted it from the drive management (this does not destroy the data on the drive, just detaches it apparently). Reboot
  3. Manually unmount mount in /mnt (in my case, “umount /mnt/storage1″ – this depends on your setup). Reboot. At this point the partition should not be in use by FreeNAS
  4. Time to fix the partition. I ran “fsck -t ufs -y /dev/ad4p1″ several times, until no errors appeared. Reboot
  5. Via the FreeNAS web interface, added the drive (making sure pre-formatted was set to UFS). Re-added the share

That’s it. FreeNAS is up, running and stable… until the next power outage that is.