openSeaChest
openSeaChest copied to clipboard
Seagate
ST16000NM000J will not --setSectorSize 4096
A handful of ST16000NM000J-2TW103 drives, which declare the "Set Sector Configuration" Feature, are failing to --setSectorSize 4096. The Set Sector Configuration Ext commands return ABORTED. Single-user Centos 7, Supermicro chassis. Full --deviceInfo below.
Did try the latest SeaChest_Lite, which behaved the same.
# ./openSeaChest_FormatUnit -v 4 --setSectorSize 4096 --confirm this-will-erase-data -d /dev/sdh
[ cut ]
Sending ATA Set Sector Configuration Ext
Sending SAT ATA Pass-Through Command:
Protocol: NON-Data
Data Direction: No Data
Task File Registers:
[FeatureExt] = C8h
[Feature] = D7h
[CountExt] = 00h
[Count] = 01h
[LBA Lo Ext] = 00h
[LBA Lo] = 00h
[LBA Mid Ext] = 00h
[LBA Mid] = 00h
[LBA Hi Ext] = 00h
[LBA Hi] = 00h
[DeviceHead] = A0h
[Command] = B2h
CDB:
0 1 2 3 4 5 6 7 8 9 A B C D E F
0x00 85 07 20 C8 D7 00 01 00 00 00 00 00 00 A0 B2 00
Sending command with send_IO
SG IO Issued as Indirect IO
SG Masked Status = 01h - Check Condition
SG Driver Status = 08h - Driver Sense Data Available
Sense Data Buffer:
0 1 2 3 4 5 6 7 8 9 A B C D E F
0x00 72 0B 00 00 00 00 00 0E 09 0C 01 04 00 00 00 00
0x10 00 00 00 00 A0 51
Sense Key: Bh = Aborted Command
ASC & ASCQ: 0h - 0h = No Additional Sense Information
FRU: 0h = No Additional Information
Return Task File Registers:
[Error] = 04h
[Count Ext] = 00h
[Count] = 00h
[LBA Lo Ext] = 00h
[LBA Lo] = 00h
[LBA Mid Ext] = 00h
[LBA Mid] = 00h
[LBA Hi Ext] = 00h
[LBA Hi] = 00h
[Device] = A0h
[Status] = 51h
Command Time (ms): 35.65
Set Sector Configuration Ext returning: ABORTED
Failed to set sector size!
Setting the sector size to 512 succeeds:
Sending ATA Set Sector Configuration Ext
Sending SAT ATA Pass-Through Command:
Protocol: NON-Data
Data Direction: No Data
Task File Registers:
[FeatureExt] = F2h
[Feature] = 93h
[CountExt] = 00h
[Count] = 00h
[LBA Lo Ext] = 00h
[LBA Lo] = 00h
[LBA Mid Ext] = 00h
[LBA Mid] = 00h
[LBA Hi Ext] = 00h
[LBA Hi] = 00h
[DeviceHead] = A0h
[Command] = B2h
CDB:
0 1 2 3 4 5 6 7 8 9 A B C D E F
0x00 85 07 20 F2 93 00 00 00 00 00 00 00 00 A0 B2 00
Sending command with send_IO
SG IO Issued as Indirect IO
SG Masked Status = 01h - Check Condition
SG Driver Status = 08h - Driver Sense Data Available
Sense Data Buffer:
0 1 2 3 4 5 6 7 8 9 A B C D E F
0x00 72 01 00 1D 00 00 00 0E 09 0C 01 00 00 00 00 00
0x10 00 00 00 00 A0 50
Sense Key: 1h = Recovered Error
ASC & ASCQ: 0h - 1Dh = ATA Passthrough Information Available
FRU: 0h = No Additional Information
Return Task File Registers:
[Error] = 00h
[Count Ext] = 00h
[Count] = 00h
[LBA Lo Ext] = 00h
[LBA Lo] = 00h
[LBA Mid Ext] = 00h
[LBA Mid] = 00h
[LBA Hi Ext] = 00h
[LBA Hi] = 00h
[Device] = A0h
[Status] = 50h
Command Time (ms): 34.32
Set Sector Configuration Ext returning: SUCCESS
Successfully set sector size to 512
And for completeness:
==========================================================================================
openSeaChest_Format - openSeaChest drive utilities - NVMe Enabled
Copyright (c) 2014-2021 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
openSeaChest_Format Version: 2.3.1-2_2_3 X86_64
Build Date: Jun 21 2021
Today: Thu Mar 24 18:07:00 2022 User: root
==========================================================================================
/dev/sg7 - ST16000NM000J-2TW103 - ZRS0BPPE - ATA
Model Number: ST16000NM000J-2TW103
Serial Number: ZRS0BPPE
Firmware Revision: SN02
World Wide Name: 5000C500E36B9F0E
Drive Capacity (TB/TiB): 16.00/14.55
Native Drive Capacity (TB/TiB): 16.00/14.55
Temperature Data:
Current Temperature (C): 42
Highest Temperature (C): 42
Lowest Temperature (C): 42
Power On Time: Not Reported
Power On Hours: Not Reported
MaxLBA: 31251759103
Native MaxLBA: 31251759103
Logical Sector Size (B): 512
Physical Sector Size (B): 4096
Sector Alignment: 0
Rotation Rate (RPM): 7200
Form Factor: 3.5"
Last DST information:
DST has never been run
Long Drive Self Test Time: 22 hours 52 minutes
Interface speed:
Max Speed (Gb/s): 6.0
Negotiated Speed (Gb/s): 6.0
Annualized Workload Rate (TB/yr): 0.00
Total Bytes Read (KB): 131.07
Total Bytes Written (B): Not Reported
Encryption Support: Not Supported
Cache Size (MiB): 256.00
Read Look-Ahead: Enabled
Write Cache: Enabled
Low Current Spinup: Disabled
SMART Status: Unknown or Not Supported
ATA Security Information: Supported
Firmware Download Support: Full, Segmented, Deferred
Specifications Supported:
ACS-4
ACS-3
ACS-2
ATA8-ACS
ATA/ATAPI-7
ATA/ATAPI-6
ATA/ATAPI-5
SATA 3.3
SATA 3.2
SATA 3.1
SATA 3.0
SATA 2.6
SATA 2.5
SATA II: Extensions
SATA 1.0a
ATA8-AST
Features Supported:
Sanitize
SATA NCQ
SATA Software Settings Preservation [Enabled]
SATA Device Initiated Power Management
Power Management
Security
SMART [Enabled]
48bit Address
PUIS
GPL
Streaming
SMART Self-Test
SMART Error Logging
Write-Read-Verify
DSN
AMAC
EPC [Enabled]
Sense Data Reporting
SCT Write Same
SCT Error Recovery Control
SCT Feature Control
SCT Data Tables
Host Logging
Set Sector Configuration
Storage Element Depopulation
Seagate In Drive Diagnostics (IDD)
Adapter Information:
Vendor ID: 1000h
Product ID: 0097h
Revision: Not available.
Hi @CraigLoomis,
Thanks for the report. Can you share the verbose output of the whole process?
Part of the set sector configuration command is to set a unique identifier from log page 2Fh in the command and I want to verify that it read that page and the identifier correctly when issuing the command to the drive to verify that everything is setup correctly.
Example: openSeaChest_Format -d <handle> --setSectorSize 4096 -v 4 | tee verboseSetSecSize.txt
The part I need would be right before the set sector configuration command in your current output.
Thanks for the file. I do not see anything setup wrong in the commands, so I do not know why the command returned an abort status.
This seems similar to Seagate/Toolbin#23 where they had a out of band management module interrupting the command with a reset, but the response from the drive in the file you shared does not match what I would expect and looks more like the drive rejected the command.
Can you try power cycling the drive and trying it again?
No change after a long power cycle. All six of the drives in the group show the same behavior: comparing these logs from two of them show diffs in some returned buffers (some changes obviously serial numbers, counters, etc) and in the elapsed times, but nothing "structural". Am now back to multiuser mode. I don't see reason to suspect occasional OOB disruption.
That said, the disks are behind an LSI 3008 SAS controller. Are SATA commands or responses liable to be modified at this level of granularity?
@CraigLoomis, Thanks for trying that. Let me check with someone internally to figure out what is going on since nothing looks suspicious in the output. You may want to search your serial number on https://www.seagate.com/support/downloads/ to see if there is a firmware update available. This may correct the issue.
That said, the disks are behind an LSI 3008 SAS controller. Are SATA commands or responses liable to be modified at this level of granularity?
Some controller firmware will block commands and some OS drivers do as well, but I do not think that either of these are happening here. I've only observed controller firmware blocking commands in RAID mode, but that was limited to security commands. I have not observed any other commands to be blocked. Blocked commands are rejected with an error code, not necessarily a drive response like your output shows. I have seen controllers send resets when something on the bus happens that makes the controller firmware suspect the drive is hung and the controller tries to recover it. I cannot tell if that is happening here since it looks like a standard abort status from the drive. When I've observed resets interrupting commands, the sense data comes back with other completely bogus information that does not even remotely look like a standard command abort response. In the other issue I linked previously, there was an issue with an out-of-band management system resetting the drive, likely for similar reasons that it thought something was wrong while the operation was running.
The firmware is up-to-date, I believe (SN02). One protocol question: the command log shows [DeviceHead] = A0h in the Set Sector Configuration Ext command block. But the manual shows both those set bits as "Obsolete". That value is used all over the place in ata_cmds.c, so a) I doubt that is significant and b) I'm hesitant to change the #define to 0x00. But still: would there be any value is just changing it in ata_Set_Sector_Configuration_Ext?
I asked our spec representative and those bits are not needed on current products and were only needed for PATA/IDE. He also confirmed that SATA devices will ignore them without generating errors. I will be creating an internal task to only set them for PATA drives and certain backwards compatible scenarios, but it does not sound like these would create an error. If you want to change that definition to a value of zero to test for yourself, that should be fine and should not affect the drive in any negative way. The old specs just said to set these to 1 without any additional explanation. I think this was to help mitigate noise on the parallel cables.
I've been looking into the cause as best I can. Today I submitted a firmware ticket to have them investigate since there does not seem to be anything obvious and quick to fix the issue from what I can tell after talking to one of the firmware engineers.
Tried 0x00 in the #define: no change in behavior. There were a few changes between the 2.3.1-2_2_3 and 2.3.4-2_4_0 versions but they do not look interesting to this.
I appreciate your knowledge and effort on this. I filed an issue at the openSeaChest github project because for groups like ours that is the more familiar and comfortable place to do so. And it turns out that you clearly have direct access to exactly the right Seagate engineering contacts to address problems like this. Assuming this is actually within your remit I congratulate Seagate for providing this less traditional channel.
Given what you have seen, do you think it would be worth trying from a different machine? Still Linux so this same program, but direct SATA. [This is at a remote site with no dedicated support staff, else I would have tried earlier.]
Given what you have seen, do you think it would be worth trying from a different machine? Still Linux so this same program, but direct SATA. [This is at a remote site with no dedicated support staff, else I would have tried earlier.]
If this is possible, yes. I know this isn't always easy in servers or in places where support is difficult, but if you can test one and see if it works, it will help narrow down what the issue might be.
If it fails, can you share information about this machine? openSeaChest and SeaChest will output PCIe vendor/product IDs at the bottom of -i if you do not have all the details. I can usually find a way to look this up and see what it is.
If the other machine works, would you please let me know? Also, would you be willing to share more details about the server that was failing? I know you shared some controller info, but any other backplanes, interposers, adapters that might be in use would be helpful to figure out what the issue might be.
Sorry I do not have a good answer from our firmware team on a potential cause as that is still being investigated from their end.
I was only able to run it using a USB adapter, running on the same machine. The command ran for a minute and a half and reported "Successfully set sector size to 4096", but per --showSupportedFormats the configuration was not changed.
The motherboard is a Supermicro X11DPH-T with a SAS3008 Fusion-MPT SAS controller for the drives.
sas3008.txt setSectorSize_4096_sdh_usb_1.txt showSupportedFormats_sdh_usb.txt
Hi @CraigLoomis,
What sector size shows up with -i?
When I reviewed the output in the showSupportedFormats file, the identify raw data seems to indicate it is running at 4k, so this may be a bug in the output for showing supported sizes.
What shows up with -i --SATInfo?
In this mode it will show 2 reports. The first is the SCSI translated info (which may have cached the original sector size) and the ATA passthrough information separately.
The important report will be from the ATA passthrough section which will display second.
Well well! Logical Sector Size (B): 4096 in the ATA section. So completely selfishly I can call this solved?
I had not appreciated how complex/non-transparent the layering is. Besides this, I noticed that the USB adapter rewrote the device serial number, which seems like an odd thing to do.
Do you think that 3008 card is doing something strange/wrong?
Excellent news!
Yeah, it is a bit more complicated than most people realize. For anything outside of a AHCI card, like the 3008 or a USB adapter, the host operating system communicates with SCSI commands as that is how the device shows up. T10 (SCSI committee) writes a spec known as SAT (SCSI to ATA translation) that describes how the adapter (USB bridge, HBA firmware, even drivers or the OS) are supposed to perform the translations to make a SATA device look like a SCSI device to the host when it issues the SCSI commands. Libata does this in Linux for SATA and PATA drives connected to AHCI or PATA adapters and is why you can use the sg3utils, lsscsi, and sdparm on SATA drives...at least for the implemented translations. One of the commands in the SAT spec is how to pass through specific commands to the target ATA drive and that is what SeaChest tools use for most of how it works. HBA firmware's are usually very good and up to date with translations, but USB adapters are generally very scarce on what they support and so much of what this tool does is so specific to a drive feature that this is used for most communication.
It is not uncommon for USB adapters to implement modified translations either, especially with regards to the model number and serial number information.
openSeaChest does not have a good clean way to determine when to depend on SCSI or ATA reported information. The default output to -i is a combination of both sets of information, relying on certain things to come from the SCSI data and others to come from the ATA data. This is partly to do with what translations USB adapters implement and partly with how to deal with products Seagate sells as USB drives.
One thing I have never found a good solution to is how to update the cached drive information that returns from the SCSI commands (such as the sector size). So far the best answer I have is a reboot or in the case of USB unplug it and plug it back in.
Do you think that 3008 card is doing something strange/wrong?
I am not sure in your case. I have one of these cards in one of my test systems (9300-8i and a 9400-8i) and have used RAID versions and have not seen this happen with this command before. It is possible that this one is doing something different as maybe it's a special supermicro version, but I do not know for sure.
There are some scenarios I have observed problems with HBA firmware in the past, but those were with older models that were fixed by updating to newer HBA firmware versions (I think we reported one to the HBA manufacturer). I did not observe the issue on SATA drives, but SAS drives and only for one specific vendor unique command we use in some internal tools. I have observed on one specific RAID model that a command was blocked, but that does not seem to be a common thing to do from the HBA firmware level. Blocked commands in the OS are more common (Windows mostly).
Does this system have an out of band management service/system that is running? We have heard of those causing interruptions. I'm looking into a few other ideas that can help capture more information to debug what is going on exactly and causing issues, but do not have anything to implement or change yet.
I'm glad your USB adapter worked, but I would like to figure out if there is something that can be done to get it working with your HBA so you do not need to remove the drives from the system to do this.
No daemon running, and to report for this Issue I ran from single-user mode in any case.
FWIW, the reported versions are LSISAS3008: FWVersion(16.00.10.00), ChipRevision(0x02), BiosVersion(18.00.00.00).
I am going to have to turn these disks into actual storage pretty soon. If there are any further tests you would like me to make please ask.
Thanks for all the information to help debug this!
I found the Seagate engineer that works with SuperMicro and asked for his help to try to figure this out. He was unable to repeat it on the systems he had which use the same 3008 HBA, but none were the X11 model like yours. So he reached out to his contact at SuperMicro to see if they could do a test and see if they could repeat it. The SuperMicro lab could not repeat it either on their X11 with a 3008 HBA.
So I am really not sure what is going on in your case right now. I wish I had better news than this. I still have an internal issue setup with our firmware team to help look into this, but it seems like there is some variable we cannot get to match your setup right now.
I do not know what other tests we can do at this point or what other system information or configuration information to ask for either. If you think of anything that may stand out that you think may be helpful, I will do the best I can to continue looking into it. I'm also continuing to look into what other debug type code that might help to assist debugging further, but not sure when I will find something worth trying right now.
I know you need to get to using these drives, but if you end up thinking of anything that might help us repeat it, please let us know.
The https://github.com/Seagate/openSeaChest/releases/tag/v23.03 release incorporates the USB fix in the earlier commit and we've added more warnings around the --setSectorSize option and made the automatic recovery better when the tool is able to detect an issue and send the vendor unique Seagate quick format command to recover the drive.
I'm closing this issue since we have done all the things we possibly can to prevent problems and recover from them that are possible in software at this point. The enhanced warnings are the last thing we were able to add to try informing customers of things that may cause issues when running this command that they may be able to mitigate a bit better.
Please feel free to create new issues if you run into anything else in openSeaChest or SeaChest and we will do our best to assist however we are able to!