Yesterday, after about 40K hours of uptime, the HDD behind /volume3 on my NAS has died.

It didn’t go “poof”, but its health got bad enough for my NAS to warn me. The advice was plain and simple: backup everything and get rid of the crashed volume.

Fortunately, this was one of the volumes containing less valuable data so I didn’t lose anything important. I’ve also got local and remote backups of the more important things.

Still, losing a disk is never fun and leads to a lot of wasted time. After a few hours, I could recover most of the data on the disk apart from a few files lying across bad sectors.

Then, just out of curiosity I wanted to check the disk and try a repair of the volume.

First, I’ve shut down every service apart from the SSH daemon:

syno_poweroff_task -d

Then, I’ve identified the faulty disk/RAID array using the commands I’ve shared in a previous post:

cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md3 : active raid1 sde3[0]
      3902296256 blocks super 1.2 [1/1] [U]

md6 : active raid1 sdc3[0]
      3902296256 blocks super 1.2 [1/1] [U]

md5 : active raid1 sdf3[0]
      3902296256 blocks super 1.2 [1/1] [U]

md7 : active raid1 sdg3[0]
      3902296256 blocks super 1.2 [1/1] [U]

md2 : active raid1 sda3[0]
      3902296256 blocks super 1.2 [1/1] [U]

md9 : active raid1 sdh3[0]
      7809204416 blocks super 1.2 [1/1] [U]

md8 : active raid1 sdd3[0]
      3902196416 blocks super 1.2 [1/1] [U]

md4 : active raid1 sdb3[0](E)
      1948792256 blocks super 1.2 [1/1] [E]

md1 : active raid1 sda2[0] sdb2[1] sdc2[2] sdd2[3] sde2[4] sdf2[5] sdg2[6] sdh2[7]
      2097088 blocks [8/8] [UUUUUUUU]

md0 : active raid1 sda1[0] sdb1[2] sdc1[4] sdd1[6] sde1[1] sdf1[3] sdg1[5] sdh1[7]
      2490176 blocks [8/8] [UUUUUUUU]

unused devices: 

As you can see above, the array in error was md4 with the associated sdb3 disk.

NOTE: I only have single-drive RAID “arrays”.

Then I took a look at the md4 array:

mdadm --detail /dev/md4
/dev/md4:
        Version : 1.2
  Creation Time : Sun Sep  8 10:16:10 2013
     Raid Level : raid1
     Array Size : 1948792256 (1858.51 GiB 1995.56 GB)
  Used Dev Size : 1948792256 (1858.51 GiB 1995.56 GB)
   Raid Devices : 1
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Wed Dec 26 21:40:05 2018
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : NAS:4  (local to host NAS)
           UUID : 096b0ec0:3aec6ef5:5f685a2b:5ff95e38
         Events : 7

    Number   Major   Minor   RaidDevice State
       0       8       19        0      active sync   /dev/sdb3

And at the disk:

mdadm --examine /dev/sdb3
/dev/sdb3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 096b0ec0:3aec6ef5:5f685a2b:5ff95e38
           Name : NAS:4  (local to host NAS)
  Creation Time : Sun Sep  8 10:16:10 2013
     Raid Level : raid1
   Raid Devices : 1

 Avail Dev Size : 3897584512 (1858.51 GiB 1995.56 GB)
     Array Size : 3897584512 (1858.51 GiB 1995.56 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 46ed084c:686ee160:5fa3a986:574d1182

    Update Time : Wed Dec 26 21:40:05 2018
       Checksum : 9ffab586 - correct
         Events : 7


   Device Role : Active device 0
   Array State : A ('A' == active, '.' == missing)

udevadm info --query=all --name=/dev/sdb3
P: /devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sdb/sdb3
N: sdb3
E: DEVNAME=/dev/sdb3
E: DEVPATH=/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sdb/sdb3
E: DEVTYPE=partition
E: ID_FS_LABEL=NAS:4
E: ID_FS_LABEL_ENC=NAS:4
E: ID_FS_TYPE=linux_raid_member
E: ID_FS_USAGE=raid
E: ID_FS_UUID=096b0ec0-3aec-6ef5-5f68-5a2b5ff95e38
E: ID_FS_UUID_ENC=096b0ec0-3aec-6ef5-5f68-5a2b5ff95e38
E: ID_FS_UUID_SUB=46ed084c-686e-e160-5fa3-a986574d1182
E: ID_FS_UUID_SUB_ENC=46ed084c-686e-e160-5fa3-a986574d1182
E: ID_FS_VERSION=1.2
E: ID_PART_ENTRY_DISK=8:16
E: ID_PART_ENTRY_NUMBER=3
E: ID_PART_ENTRY_OFFSET=9437184
E: ID_PART_ENTRY_SCHEME=dos
E: ID_PART_ENTRY_SIZE=3897586881
E: ID_PART_ENTRY_TYPE=0xfd
E: ID_PART_ENTRY_UUID=00003837-03
E: MAJOR=8
E: MINOR=19
E: PHYSDEVBUS=scsi
E: PHYSDEVDRIVER=sd
E: PHYSDEVPATH=/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0
E: SUBSYSTEM=block
E: SYNO_DEV_DISKPORTTYPE=SATA
E: SYNO_INFO_PLATFORM_NAME=cedarview
E: SYNO_KERNEL_VERSION=3.10
E: USEC_INITIALIZED=112532

Then I’ve unmounted the faulty volume and stopped the corresponding RAID array:

umount /volume3
mdadm –stop /dev/md4

After that, I’ve re-created the array:

mdadm -Cf /dev/md4 -e1.2 -n1 -l1 /dev/sdb3 -u096b0ec0:3aec6ef5:5f685a2b:5ff95e38

Finally, I ran a file system check:

fsck.ext4 -v -f -y /dev/mapper/vol3-origin

Where /dev/mapper/vol3-origin was an easy to use pointer to the device.

From the NAS’s point of view, everything is now fine (haha), but of course I can’t trust that disk anymore. Now I just have to wait a few days to get a replacement and set it up.

On the bright side, I’ll take the occasion to upgrade to a 10-12TB disk (assuming those are compatible with my Synology NAS..) ^_^. That way I’ll prepare a new disaster for today + 40K hours.. ;-)

I’ve published a new post on Medium:

https://medium.com/@dSebastien/the-story-behind-my-upcoming-book-learn-typescript-by-building-web-applications-part-1-26926bd1756d

I’ve just published a new article on Medium.com.

I’m lazy today, so I’ll just give you a link to it:
https://medium.com/@dSebastien/using-junit-5-with-spring-boot-2-kotlin-and-mockito-d5aea5b0c668

Enjoy!

About

Some notes about my current setup for GPG/PGP.

I’m currently using GnuPG: https://www.gnupg.org/ and in particular GPG4Win.

Portable mode

As usual, I like portable installs and GPG is no exception. I’ve uncompressed it in my tools folder (synchronized across my machines). By itself, the tool is portable, maybe Kleopatra isn’t but I don’t care too much.

By default, Gpg4win installs in two locations:

• Gpg4win: C:\Program Files (x86)\Gpg4win
• GnuPG: C:\Program Files (x86)\GnuPG

Bash profile

Here’s how my bash profile is configured to have GPG tools available:

# GPG/PGP
# where the tool is installed
export GPG4WIN_HOME=$TOOLS_HOME/Gpg4Win_3.0.1
export GPG_HOME=$GPG4WIN_HOME/GnuPG
export KLEOPATRA_HOME=$GPG4WIN_HOME/Gpg4win

append_to_path $GPG_HOME
append_to_path $GPG_HOME/bin
append_to_path $KLEOPATRA_HOME/bin_64
append_to_path $KLEOPATRA_HOME/bin

# where it puts its files and looks for its configuration
export GNUPGHOME=$HOME/.gnupg

# create it otherwise it complains
mkdir -p `echo $GNUPGHOME`
alias gpg='gpg.exe'
alias pgp='gpg' # who cares ;-)
alias kleopatra='kleopatra.exe'

GPG configuration

Here’s my current GPG configuration (~/.gnupg/gpg.conf). I’ve removed comments for stuff I don’t use for clarity, although I like to keep those in my actual configs):

# get rid of the copyright notice
no-greeting

# key server
keyserver hkp://keys.gnupg.net

# Ensure that stronger secure hash algorithms are used by default
default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAMELLIA256 CAMELLIA192 CAMELLIA128 TWOFISH CAST5 ZLIB BZIP2 ZIP Uncompressed
personal-digest-preferences SHA512
cert-digest-algo SHA512
 
# Enable the passphrase agent
use-agent
 
# Avoid locking files
lock-never

# Armor when exporting
armor
 
# Keyserver options
keyserver-options auto-key-retrieve include-subkeys honor-keyserver-url honor-pka-record
 
# Import/export options
import-options import-clean
export-options export-clean

# Don't use key ids are those are unsafe (both short and long!)
keyid-format none

With this configuration, I’ve forced the usage of stronger secure hash algorithms by default and also disabled key ids (short & long) since those are insecure. There’s nothing much to it.

How I generated my keys

First of all, I didn’t reinvent the wheel, I’ve mostly applied what Alex Cabal has described here, so thanks to him!

You might ask “Why not a simple key that does it all?”. Because in general, mixing signing and encryption keys is not a good idea, management & security wise. Firstly, different key types have different lifecycles. Secondly, it might just not be safe to do so.

Also, without this setup, if the keys I use on a daily bases were to be compromised, I wouldn’t have any other choice but to re-create everything from scratch (i.e., new identity!). With the configuration below I can just revoke a specific sub-key and create a new one, while keeping my identity.

Here’s the whole shabang.

Create the keypair

First of all, create the key:

gpg --gen-key

Settings to use:

• Kind of key: (1) RSA and RSA
• Key size: 4096 (longer = safer?)
• Valid for: 0 (never expires)
• mail: [email protected]

When selecting the passphrase, use a tool like Keepass, don’t choose the passphrase yourself, you’re not smart enough ;-).

Set strong hash preferences on the keypair

Just to make sure:

gpg --edit-key [email protected]
...
gpg> setpref SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAMELLIA256 CAMELLIA192 CAMELLIA128 TWOFISH CAST5 ZLIB BZIP2 ZIP Uncompressed
gpg> save

Add a signing sub-key

Next, create a signing sub-key for code signing:

gpg --edit-key [email protected]
...
gpg> addkey
...
gpg> save

Settings:

• Key type: (4) RSA (sign only)
• Key size: 4096 (longer = safer?)
• Valid for: 0 (never expires)
• mail: [email protected]

Add an authentication sub-key

Next, create an authentication sub-key for SSH authentication:

gpg --expert --edit-key [email protected]
gpg>addkey
...
gpg>save

Settings:

• (8) RSA (set your own capabilities)
• S: disable sign
• E: disable encrypt
• A: enable authenticate
• –> now you must see “Currently allowed actions: Authenticate”
• Q: finished
• Key size: 4096
• Expires: today + 365 days

Create a revocation certificate

Generating a revocation certificate will allow me to later revoke this keypair if it is compromised. It must be kept safe because it can render my keys useless ;-)

gpg --output ./[email protected] --gen-revoke [email protected]

Export the keypair/subkeys to a safe location and make the key safe to use

First export the private key:

gpg --export-secret-keys --armor [email protected] > [email protected]

Then export the public key:

gpg --export --armor [email protected] > [email protected]

Finally, you can export the sub-keys alone:

gpg --export-secret-subkeys [email protected] > /tmp/gpg/subkeys

We’ll see why afterwards.

Ideally, you should export your private key to a temporary in-memory file system. Alex proposed the following:

mkdir /tmp/gpg # create a temp folder
sudo mount -t tmpfs -o size=1M tmpfs /tmp/gpg

Once that’s mounted, you can safely write there and remove the folder once you’re done.

Once exported, back-up those keys in a safe location (e.g., Keepass).

Once you’re 100% it’s backed-up, delete the secret key from the gpg keyring:

gpg --delete-secret-key [email protected]

Now re-import the sub-keys. With this you’ll only have the sub-keys at your disposal (and you don’t need more than that on a daily basis):

gpg --import /tmp/gpg/subkeys

So simple steps:

• create/mount the temporary in-memory file system
• export your private key
• back it up in a safe location
• remove the temporary file system
• bonus: burn the machine you’ve done this upon ;-)

To verify that you didn’t mess up, go ahead and try to add a new sub-key; you shouldn’t be able to:

gpg --edit-key [email protected]
gpg> addkey
Secret parts of primary key are not available.
gpg: Key generation failed: No secret key

That’s it!

How I can revoke a sub-key

Using Google! Err I mean like this: https://wiki.debian.org/Subkeys.

First re-import my whole key (i.e., master + sub-keys)

gpg --allow-secret-key-import --import 

Second, edit the key and revoke the sub-key that I don’t want anymore:

gpg --edit-key [email protected]
gpg> list # list the keys
gpg> xyz # select the unwanted key
gpg> revkey # generate a revocation certificate
gpg> save

Once done, I can export/back-up the result and finally make sure to send the updated key to the key servers.

Where I’ve published my key

Once my key was ready, I’ve published it at various locations.

For starters I needed the full fingerprint (the 40 chars beauty):

gpg --fingerprint

In my case: 9AEC 7595 2F0F 8E52 65A8 4364 6448 ABB4 AEAD 81A2.

Just to be in the clear, if you need to share your key, always try to use the full fingerprint, certainly never the short version (8 hex chars one) nor the “long” (16 hex chars) since those are really unsafe.

First I sent the public key to the MIT key server using gpg:

gpg --send-keys [email protected]

Then I exported my public key to a file (ASCII-armored):

gpg --export --armor > dsebastien-pgp-key.asc

I then uploaded that file to my FTP, updated my about page to add the full fingerprint and a link to my public key. Then I added a blog post with the same information.

I’ve also sent an update on twitter the same. After that I’ve updated my Twitter bio to link to that tweet (optimizing character count :p).

Next up, I’ve uploaded the public key manually on Ubuntu’s key server.

Finally, I’ve updated my GitHub profile to add my PGP key.

Git client configuration

I’ve also updated my git client configuration in order to make my life easier.

• git config –global user.signingkey 9AEC75952F0F8E5265A843646448ABB4AEAD81A2

This tells git which key to use. BTW, don’t enable automatic commit signing. Sign tags instead.

Verifying signatures is a breeze with git.

Later

In a later post, I’ll explain how I use my PGP keys with SSH, git and my Yubikey.

That’s all folks!

I’ve got a new PGP key.

My PGP key fingerprint is: 9AEC 7595 2F0F 8E52 65A8  4364 6448 ABB4 AEAD 81A2

You can find my public PGP key here: https://dsebastien.net/pgp/dsebastien-pgp-key.asc