1. Home
  2. Knowledge Base
  3. Security
  4. NetApp NSE Setup and Management

NOTE: NSE must be ALL or NONE NSE per HA-Pair

Authentication Keys (AK) and changes to them do not affect the disk encryption keys
When a system is first brought up, the NSE disks are openly available to the system without need for authentication.  The disks themselves automatically encrypt data written to them and decrypt it when read and maintain these disk encryption keys (AKA media encryption keys) within themselves.  The controls are not yet set to protect a disk that leaves the system.  The system may be operated in this unprotected mode indefinitely.  The NSE disks simply act like other disks.

When the servers are made available and the required SSL/TLS certificates are properly installed, the setup of the connections between the KMIP servers and the cluster is made.  Thereafter, authentication keys can be created and the controls in the disks set to protect the data.  Then, if the disks are power-cycled, such as would happen if a disk is removed and placed on another system, that system cannot give the required AK (safely on an SSL-protected key server) to unlock access to the data.

Modifying authentication keys does not affect the encryption keys.  Data that is written to the disks in the period before KMIP server setup and AK changes is still present.  Once the controls are set, then all data on the disks is protected, whether it existed before or after the protections were applied.

The disks come with a default key, called the Manufacture Secure ID (MSID), that is unique to each disk.  It is electronically readable from the disk, so it provides no protection on its own.  This might be what the questioner referred to as “the open key.” When Data ONTAP modifies the AK to a new value the MSID can no longer be used to access the disks, if it should leave the system.

Storage encryption is at the disk firmware on self-encrypting disks (SEDs).  SEDs run in unprotected or protected mode (encrypted). Protected mode requires key manager authentication after power-on.  There is no noticeable performance decrease or boot time increase.  Furthermore, all Data ONTAP storage efficiencies (i.e. dedupe, compression) are not affected.

You can specify up to 4 key servers during or after setup.  You should have at least 2 key servers.  If you have production and DR site the key managers are clustered together; this is a common setup.

SEDs have two additional features in addition to encryption

  1. Sanitize (for return) changes the encryption key to a new unknown key
  2. Destroy changes the encryption key to a random unknown key and permanently locks the disk
  • Gemalto SafeNet Professional Services are REQUIRED for installation of the key manager
  • Certifications and reasons for NSE
    • FIPS 140-2 level 2
    • PCI
    • HIPAA
    • FISMA
    • CA SB1386
  • cDOT 8.2.1+ (7-Mode 8.1.1+)
    • 8.2.1 disk encrypt commands are run at node shell
    • 8.3+ disk encrypt commands MUST be run at cluster shell
  • When a drive is replace by RMA, make sure to run the storage encryption disk modify command
  • FIPS compliance mode 8.3.1+ ONLY
    • FIPS key can be different from the drive authentication key
    • Sanitizing or resetting to factory settings removes FIPS compliance mode
    • If set to FIPS compliance and downgrade <8.3.1, disable FIPS on all SEDs storage encryption disk modify -disk * -fips-key-id 0x0
  • KMIP 1.0 and 1.1 supported
  • 256-bit AES encryption depending on drive
  • No MetroCluster support
  • Do NOT use 10Gb interfaces for KMIP
    • e0M is recommended because different models might not have other interfaces available during the boot process
  • Cluster Integration
    • 1 authentication key per node
    • Each disk has its own DEK (Drive Encryption Key)
  • Up to 128 Authentication keys per Cluster with a warning threshold of 100

NOTE: See the Interoperability Matrix for supported key management servers.  External Key Management is REQUIRED with NSE for Data ONTAP <9.0.  Data ONTAP 9.0 has onboard key management.

How to determine whether you need an external key management server

You should set up an external key management server if any of the following are true for your environment:

  • Your encryption key management solution must comply with Federal Information Processing Standards (FIPS) 140-2.
  • You need a multi-cluster solution.  With an external key management server, you can have a single key server solution that supports multiple clusters with centralized management of all encryption keys.
  • You need support of industry OASIS Key Management Interoperability Protocol (KMIP) standard.
  • Your business requires the added security of storing your encryption keys on a system or in a location different from the data.  With onboard key management, the data and the encryption keys are stored on the same system.  With an external key server, the data and keys are stored separately.

Current Support (subject to change)

  • Gemalto SafeNet KeySecure k460
    • 30K list
    • Physical
    • FIPS-140-2 level 3
    • 1 million keys
    • LDAP, AD, Syslog forward
  • Gemalto SafeNet KeySecure k150v
    • 10K list
    • Virtual
    • FIPS-140-2 level 1
    • 25,000 keys (100 per NetApp cluster)
    • LDAP, AD, Syslog forward
  • IBM TKLM/SKLM (not sold by NetApp)
  • Vormetric DSM (PVR ONLY and not sold by NetApp)

Setting Up NSE

SSL

  • Authentication cluster to KMIP is over SSL and we need both public and private SSL certificates for the cluster and the public SSL certificate of the KMIP server.
    • Synchronize the time between all devices including the server creating the certs
    • SSL client cert must not be password protected
    • Common name of certs can not contain “_” underscore
    CertFilenameNotes
    HA-Pair Publicclient.pem
    HA-Pair Privateclient_private.pemcan be a passphrase
    KMIP Publickeymanagementserver_ip.CA.pem

Install Certificates on the cluster

SSL client certificate for the cluster

::> security certificate install -vserver svm_name -type client -subtype kmip-cert

  • Prompts for the public and private key

Install public certificate for the root certificate authority (CA) on the SSL server (external KMIP server)

::> security certificate install -vserver svm_name -type server-ca -subtype kmip-cert -kmip-server ip

  • Enter IP by subnet if multiple, for example 192.168.150.10, 192.168.150.11, 192.168.150.12 use “192.168.150.0” or use “0.0.0.0” as a catch all
  • If different root CAs, repeat the command for each KMIP public certificate SSL certificates expire.  Remove old certs before installing new certs. 

Remove old certs

::> security key-manager delete -address x.x.x.x # from key management server
::> security certificate delete -type client -vserver svm_name -common-name fqdn_or_custom_common_name -ca certificate_authority -subtype kimp-cert # from cluster's client certs
::> security certificate delete -type server-ca -vserver svm_name -common-name fqdn_or_customer_common_name -ca certificate_authority -subtype kmip-cert # remove all installed key management server certs - enter for EACH key management server

Go back to top of this sub-section and run security certificate install above for -type client and type server-ca, and then repeat security key-manager add below for additional key manager servers

Add a key manager server (best practice = 2 total; max = 4)

::> security key-manager add -address keymanager-ip

Delete management server

::> security key-manager delete -address x.x.x.x

Verify key management server links

::> security key-manager show
::> security key-manager show -status # all key managers linked
::> security key-manager show -status -address x.x.x.x.x

Setup Encryption with the key management setup wizard (SSL certificates MUST be installed). Also run this when changes are made after initial setup

::> security key-manager setup -node nodename

PromptsDefault
TCP port5696
Interface (1Gb)e0M
IPv4yes
IPx.x.x.x
Maskx.x.x.x
GWx.x.x.x
IPv6no

Disk Encryption (create-key and modify disk to key)

::> storage encryption disk show
::> security key-manager show -status

Create Key with -key-tag (=clustername by default)

    • Manual

::> security key-manager create-key -prompt-for-key true -key_tag key_tag

    • Automatic

::> security key-manager create-key -key_tag key_tag

Assign Key to Disks (and also after an RMA of a broken disk)

::> storage encryption disk modify -disk disk_id -data-key-id auth_key_id
::> storage encryption disk show # confirm key was changed

Turn ON FIPS (OPTIONAL) after disk encryption above

::> storage encryption disk modify -disk disk_id -fips-key-id fips_auth_key_id
::> storage encryption disk show-status
::> storage encryption disk show -fips

Turn OFF FIPS, Move Volumes, Reset drives)

::> storage encryption disk show -fips
::> set adv
::*> storage encryption disk modify -disk disk_id -fips-key-id 0x0

  • vol move data
  • Sanitize disks

::*> storage encryption disk sanitize -disk disk_id

  • Reset to factory original settings (if disks have PSID)

::*> storage encryption disk revert-to-original-state -disk disk_id -psid disk_physical_secure_id
::*> storage disk fail -disk disk_id -immediate true
::*> storage disk unfail -disk disk_id
::*> storage encryption disk show
::*> set admin

::> storage encryption disk show -fips

Retrieve Authentication Keys (new node or node that was halted)
When you create a new key, ONTAP automatically restores the key so the new key is available on all nodes. This is for NEW or HALTED nodes that do not have the key.

::> security key-manager restore -address x.x.x.x # restore to all nodes
::> security key-manager restore -node nodename

Return SEDs to unprotected mode (default MSID keys)

::> set adv

Disable FIPs

::> storage encryption disk show -fips
::*> storage disk encryption disk modify -disk * -fips-key-id 0x0 # only if FIPs enabled above

Change auth key to default MSID

::*> storage disk encryption disk modify -disk * -data-key-id 0x0

Remove key manager servers

::> security key-manager delete -address x.x.x.x

View and delete SSL Certs

::> security certificate show -subtype kmip-cert
::> security certificate delete -type server-ca -vserver svm_name -common-name fqdn_or_custom_commo_name -ca certificate_authority -subtype kmip-cert

  • Repeat for each KMIP server

Return to service a disk with no auth key (can’t restore data, but can re-use drive)

Two Methods

  1. NO FIPS, or FIPS key is still available

::*> storage encryption disk sanitize -disk disk-id

STOP if FIPS compliance mode and FIPS key is available. Continue if not FIPS

::*> storage disk fail -immediate true -disk disk_id
::*> storage disk unfail -spare true -disk disk_id
::*> storage disk show -disk disk_id

  1. SEDS are in FIPS compliance mode and keys are not available and IF the disk PSID is printed on the disk label

::*> storage encryption disk revert-to-original-state -disk disk_id -psid disk_physica_secure_id
::*> storage disk fail -immediate true -disk disk_id
::*> storage disk unfail -spare true -disk disk_id
::*> storage disk show -disk disk_id

Destroy / Sanitize (Shred) Before return RMA (works on spare and broken disks unless you use -force-all-states true)

::> storage encryption disk show
::> storage encryption disk sanitize -disk disk_id

  • to run against ALL drives in the cluster (be careful)

::*> storage encryption disk sanitize -force-all-states -disk *

Destroy Disk – EOL (spare and broken disks ONLY)

::> storage encryption disk destroy disk_id

  • DESTROY the PSID on the DISK LABEL and all photos/copies of the drive label

EMERGENCY SHREDDING – Destroy Disk (even if no power and no key-mgmt)

Power is Available and you have time

::> sto fail modify … # disable storage failover
::> aggr offline … # offline and destroy aggrs
::> aggr destroy …
::> halt …

Loader> boot_ontap

CTRL-C Special boot loader
Option 5 maintenance mode
::*> storage encryption disk sanitize -disk * -force-all-states true

Power is Available and you DO NOT have time

::> sto fail modify … # disable storage failover
::*> storage encryption disk sanitize -disk * -force-all-states true

  • NODES PANIC

No Power
Destroy Key Manager Keys online , or if no power on key manager destroy the smart card in the key management server

Boot a system with the key manager not available – save the passphrase from original setup
If a system is halted and moved to a new network, change the security key-manager setup before halt, or use the loader prompt to set the passphrase, then rerun security key-manager setup.  The values for the passphrase and keyid variables are documented during the Key-Manager setup process.  Once these are defined, Data ONTAP will boot.  Change the Custer and Node IPs and re-run the key-manager setup.

LOADER
unsetenv kmip.init.interface <interface>
setenv storageencryption.key.0 "<passphrase for key ID 0>"
setenv storageencryption.keyid.0 "<key ID 0>"
setenv storageencryption.last_index "1"
saveenv

boot_ontap
::> security key-manager setup -node <nodename>

OLD METHOD prior to 8.3.x. “security key-manager setup” wizard…. no boot loader needed with the cluster command method

https://kb.netapp.com/support/index?page=content&id=1012619&actp=search&viewlocale=en_US&searchid=1348162615003

How to set pre-boot environment variables for NetApp Storage Encryption using KMIP servers

KB ID: 1012619 Version: 14.0 Published date: 02/08/2016

Description

Setting pre-boot environmental variables are necessary for NetApp Storage Encryption (NSE) storage systems.  The NSE bootarg bootarg.storageencryption.support true is set at the factory if NSE drives are present in the controller.  The other bootargs listed below are automatically created when the key_manager setup (nodeshell \ 7-Mode) or the security key-manager setup (clustered Data ONTAP 8.3.1 and later) command is run.

If the storage system is unable to correctly contact the KMIP Key Management Server, it will not be able to boot.  This is because the required boot information cannot be decrypted and will therefore be unreadable.

Procedure

  1. Log in to the storage system using the service processor (SP, RLM or BMC) or through the serial console port.  This will be required for the completion of all additional steps to correctly add/verify the pre-boot settings for the KMIP server.
  2. When prompted, press Ctrl+C to halt the boot process.  This will bring up the loader prompt.  If the loader prompt does not appear, the storage system will have to be rebooted.  Ctrl+C should be used at the first prompt to halt storage system boot up, and not at the second prompt.
  3. Access the pre-boot environment variables from the loader prompt.  The command printenv will show all the existing pre-boot variables.

    Loader_A> printenvUse this command to list all pre-boot variables, and look for the following five:

    kmip.init.interface xxx
    kmip.init.ipaddr xxx.xxx.xxx.xxx
    kmip.init.netmask xxx.xxx.xxx.xxx
    kmip.init.gateway xxx.xxx.xxx.xxx
    bootarg.storageencryption.support true

    This string is required for NSE at all times. Do not remove this string or set it to false.  If the string is missing, Data ONTAP will discover the NSE hard drives as failed devices.

  4. Use the setenv command to set the variables for the pre-boot environment:

    Loader_A> setenv kmip.init.interface e0M
    (All values are case-sensitive. For example, in e0M, the M must be in upper case)

    Loader_A> setenv kmip.init.ipaddr xxx.xxx.xxx.xxx
    Loader_A> setenv kmip.init.netmask xxx.xxx.xxx.xxx
    Loader_A> setenv kmip.init.gateway xxx.xxx.xxx.xxx
    Loader_A> setenv bootarg.storageencryption.support true

    Important Notes:

    1. The four kmip.init.* values must all be set.  If any one of the values is not entered, the whole interface will be treated as unconfigured.  The interface IP used is required to be a dedicated IP address that will not be used when Data ONTAP is loaded; this is to prevent having duplicated IPs during Controller Takeover.
    2. Beginning with Data ONTAP 8.1, NSE supports FAS2040, FAS2240, FAS32xx, FAS62xx, V32xx, and V62xx.  The pre-boot environment variables applies to all NSE FAS Filers.  Depending on the FAS platform, the FAS may or may not have an e0M interface.  e0M is the recommended interface to use if the FAS platform has the management NIC.
    3. Save the changes to the pre-boot environment.  Otherwise, they will only be valid for a single boot of the filer.  Use the command saveenv to save changes to the pre-boot variables.

      Loader_A> saveenv

  5. Boot the storage system from the loader prompt. Use the following command to resume the storage system boot up:

    Loader_A> boot_ontap or bye

Was this article helpful?

Related Articles