Installing an NVME Disk on AWS EC2

So, I created i3.large with an NVME drive on each node, here is my process:

  • lsblk -> nvme0n1 (check while nvme is not installed yet)
  • sudo mkfs.ext4 -E nodiscard / dev / nvme0n1
  • sudo mount -o discard / dev / nvme0n1 / mnt / my-data li>
  • / dev / nvme0n1 / mnt / my-data ext4 by default, nofail, discard 0 2
  • sudo mount -a (check if everything is ok)
  • reboot sudo

So this all works, I can connect to the instance. I have 500 Go to my new section.

But after I stopped and restarted the EC2 computers, some of them accidentally became unavailable (AWS warning checked only for 1/2 test)

When I look at the logs about why this is not available, it tells me about this nvme section (but I did sudo mount -a to check if everything is ok, so I don’t understand)

I don't have AWS logs for sure, but I got a few lines:

Bad magic number in super block when trying to open

then the superblock is corrupted, and you can try running e2fsck with an alternative superblock:

/ dev / fd / 9: line 2: plymouth: command not found

+11
source share
3 answers

Stopping and starting an instance erases the ephemeral disks, moves the instance to the new host equipment and issues new empty disks ... so the ephemeral disks will always be empty after stopping / starting. When an instance is stopped, it does not exist on any physical node — resources are freed.

So, the best approach if you are going to stop and start instances is not to add them to /etc/fstab but simply format them on first boot and mount them after that. One way to check for a file system is to use the file utility and grep its output. If grep does not find a match, it returns false.

The NVMe SSD in the i3 instance class is an example of instance storage capacity, also known as Ephemeral [Disk | Volume | Drive unit ]. They are physically located inside the instance and are very fast, but not redundant and not intended for persistent data ... hence, “ephemeral”. Persistent data must be on the Elastic Block Volume (EBS) or the Elastic File System (EFS) , both of which can withstand shutdown / start of an instance, hardware failures, and maintenance.

It's unclear why your instances are not loading, but nofail cannot do what you expect when volumes are present but do not have a file system. I got the impression that in the end it should work out.

But you may need apt-get install linux-aws when starting Ubuntu 16.04. Support for Ubuntu 14.04 NVMe is actually not stable and is not recommended .

Each of these three storage solutions has its advantages and disadvantages.

The instance store is local, so it's pretty fast ... but, it's ephemeral. It withstands hard and soft reloads, but does not stop / start cycles. If your instance is experiencing equipment malfunctions or plans to retire, as is the case with all equipment, you will have to stop and start the instance to transfer it to new equipment. Reserved and allocated instances do not alter the behavior of the ephemeral disk.

EBS is a permanent redundant storage that can be separated from one instance and moved to another (and this happens automatically through stop / start). EBS supports snapshots of a point in time, and they increase at the block level, so you do not pay for storing data that did not change in snapshots ... but through excellent witchcraft you also do not have to track “full” and “incremental” snapshots snapshots are only logical containers of pointers to backup data blocks, so they are, in fact, “full” snapshots, but only exposed as incremental ones. When you delete a snapshot, only blocks that are no longer needed to restore that snapshot and any other snapshot are deleted from the internal storage system (which, transparently for you, actually uses Amazon S3).

EBS volumes are available in both SSD and magnetic spinning disk volumes, again with trade-offs in cost, performance and related applications. See EBS Volume Types . EBS volumes mimic regular hard drives, except that their capacity can be manually increased on demand (but not reduced) and can be converted from one type of volume to another without shutting down the system. EBS performs all data migration on the fly with reduced performance, but no failures. This is a relatively recent innovation.

EFS uses NFS, so you can mount the EFS file system as many times as you want, even in availability zones within the same region. The size limit for any single file in EFS is 52 terabytes, and your instance will actually report 8 exabytes of free space. The actual free space for all practical purposes is unlimited, but EFS is also the most expensive - if you have 52 TiB files stored there for one month, this storage will cost more than $ 15,000. The most that I have ever stored was about 20 TiB for 2 weeks, it cost me about 5 thousand dollars, but if you need a place, the space is there. It is exposed hourly, so if you saved 52 TiB files in just a couple of hours and then deleted it, you would pay maybe 50 dollars. “Elasticity” in EFS refers to power and price. You have not provided a place in EFS. You use what you need and delete what you don’t have, and the billable amount is calculated hourly.

The discussion about storage would not be complete without S3. It is not a file system, it is a repository of objects. At approximately 1/10 the price of EFS, the S3 also has virtually infinite capacity and a maximum object size of 5 TB. Some applications are best developed using S3 objects instead of files.

S3 can also be easily used by systems outside of AWS, whether in your data center or in another cloud. Other storage technologies are intended for use inside EC2, although there is an undocumented workaround that allows EFS to be used externally or through regions with proxies and tunnels.

+6
source

I had the same experience! My C5.xlarge instance detects EBS as nvme1n1. I added this line to fstab.

  /dev/nvme1n1 /data ext4 discard,defaults,nofail 0 2 

After several reboots, it looked workable. He continued to work for several weeks. But today I just got a warning that the instance was unable to connect. I tried to reboot it from the AWS console, it was unlucky that the culprit was fstab. Could not install disk.

I raised the ticket to AWS support, and there is no feedback yet. I have to start a new instance to restore my service.

In another test instance, I'm trying to use the UUID (get the blkid command) instead of / dev / nvme 1n1. While still working ... will see if it caused any problems.

I will update here if there is any support for AWS support.

================= EDIT with my fix ===========

AWS doesn't give me any feedback yet, but I found a problem. In fact, in fstab, no matter what you mount / dev / nvme 1n1 or UUID, it does not matter. My problem: my ESB has some errors in the file system. I bound it to an instance, then ran

 fsck.ext4 /dev/nvme1n1 

After fixing a couple of file system errors, put it in fstab, reboot, more problems!

0
source

You can find a useful new instance of the EC2 family equipped with NVMe local storage: C5d .

See the blog entry: https://aws.amazon.com/blogs/aws/ec2-instance-update-c5-instances-with-local-nvme-storage-c5d/

enter image description here

Some excerpts from the blog post:

  • You do not need to specify the display of the block device in your AMI or during instance startup; local memory will be displayed as one or more devices (/ dev / nvme * 1 on Linux) after loading the guest operating system.
  • In addition to adding local storage, C5 and C5d use the same specifications.
  • You can use any AMI that includes drivers for Elastic Network Adapter (ENA) and NVMe
  • Each local NVMe device is hardware encrypted using XTS-AES-256 block encryption and a unique key.
  • Local NVMe devices have the same lifetime as the instance to which they are attached, and do not close after the instance has been stopped or completed.
0
source

Source: https://habr.com/ru/post/1269943/


All Articles