sg3_utils

sg3_utils copied to clipboard

                    README for sg3_utils
                    ====================

Introduction

sg3_utils is a package of utilities originally written to send individual SCSI commands to storage devices that used one of the SCSI command sets. These utilities can be divided into three groups:

• sg_raw: the user supplies the CDB (Command Descriptor Block) and optionally the size of the data-in and data-out buffers
• one command utilities: the majority of the utilities in this package send one SCSI command. Their names start with "sg_" while the remaining part of their name alludes to the name of the command which is sent. For example, "sg_inq" sends the SCSI INQUIRY command. Some utilities in this group send one of a selection of commands, typically those commands have a lot it common (e.g. sg_write_x).
• copy type utilities: sg_dd, sgp_dd and sgm_dd use the Unix dd command functionality and command line syntax as a template. sg_dd may also be used to verify that two storage devices (or part thereof) contain the same user data, stopping at the first "miscompare". sg_xcopy sends the SCSI EXTENDED COPY command which in some cases can do offloaded copies.

Platforms

These utilities were written on Linux and should work from Linux kernel (lk) 2.4 through to the current series 6. The third group ("copy type") are only implemented on Linux, but a separate portable package/utility called ddpt implements similar functionality. The first two groups are implemented (i.e. ported) to Android, FreeBSD, NetBSD, Solaris and Windows. The Windows port uses either a Cygwin or MinGW (plus Msys) build environment (rather than Visual Studio).

Library

Many of these utilities share a lot of code (e.g. SCSI error messages) so a lot of repetition (potentially error prone) is saved by having a library called libsgutils or some variation on that name. Distributions (especially of Linux) have differing policies on how a library (and a package) should be named. For that reason this package is sometimes known as "sg3-utils" (i.e. the underscore is turned into a hyphen). Various other packages use libsgutils. The library interface is not altered from one package release, to the next, but the library interface may be expanded. If a utility from one release is used with a libsgutils from an earlier release, then the runtime linking may fail. Typically package managers take care of these details so that runtime linking errors should be rare.

Command Sets

SCSI command sets are not the only storage command sets in wide use, there are also ATA and NVMe command sets. There is a SCSI command set to translate SCSI commands to ATA commands (called SAT: SCSI to ATA Translation). SAT includes an ATA PASS-THROUGH SCSI command and sg_sat_* utilities (there are four) are examples of using SAT. The SAS transport (Serial Attached SCSI) can convey ATA commands through a SCSI/SAS domain via its Serial ATA Tunnelled Protocol (STP).

NVMe command sets (e.g. Admin, NVM and MI) are relatively new. There was an early paper on a SCSI to NVMe Translation Layer (SNTL) but it hasn't been standardized. The sg_inq utility will send (and decode the response of) a SCSI INQUIRY command if the underlying device is a SCSI device. If the underlying device is a NVMe controller or namespace, then sg_inq will send a NVMe Admin Identify command and decode the response. The sg_ses utility (for SCSI Enclosure Services) also checks whether its underlying device is SCSI or NVME. In the NVMe case, sg_ses translates the SCSI SEND DIAGNOSTIC and READ DIAGNOSTIC RESULTS commands to the NVMe Management Interface (MI) SES Send and SES Receive commands respectively. The output of the sg_ses utility should be similar, irrespective of whether the "SES" device is SCSI or NVMe.

The sg_raw utility may send NVMe Admin or NVM commands (as well as SCSI commands). One difficulty with a command-line utility invoking NVME commands is that those commands contain memory addresses for data-in (i.e. data from the storage device) or data-out (i.e. data sent toward the storage device) transfers. See the sg_raw manpage for how this difficulty is addressed.

JSON

Taking a lead from smartmontools project, JSON is slowly being added to those utilities that decode a significant amount of (meta-)data supplied by storage devices. Human readable output is still the default with the '--json' option changing the output to JSON. The '--json' option itself takes an optional argument which is a string where each character turns on (or off, if preceded by a '-') some attribute of the JSON output (e.g. '--json=h-e').

One difficulty with JSON (or the json_builder library) is that its integer representation is signed, 64 bit in size, and rendered in decimal, with a leading negative sign if needed. However storage metadata tends to use hexadecimal for large values (e.g. Logical Block Addresses (LBAs)). This issue is addressed with the 'h' argument (e.g. '--json=h') which changes decimal rendered numbers into a JSON subobject with two name-value pairs: one name is 'i' whose value is the same decimal rendered integer, and the other named 'hex' whose value is a JSON string with that integer rendered in (unsigned) hexadecimal in that string.

To simplify name decoding in JSON, the "snake notation" convention is used. This restricts names to the 26 lower case, alphabetical characters (from English), the 10 digits and underscore. Further an underscore is never the first or last character of the name and is never repeated. So underscore can be used as a name separator (e.g. "device_id"). Other JSON producing utilities follow different naming conventions. One advantage of the "snake notation" is that it is compatible with various Unix pseudo file system naming conventions (e.g. procfs and sysfs in Linux).

There is a manual page called "sg3_utils_json" that contains more information.

Documentation

Manual pages ("manpages") are the primary method of utility documentation. All utilities and scripts that are installed by this package have a manpage. There are utilities in the examples, testing and utils directories that are not installed and do not have manpages. Nearly all utilities have runtime help, usually invoked with either the '-h' short option or the '--help' long option. There is also an overarching manpage called "sg3_utils". All manpages are placed in chapter 8 which is for system administration commands/utilities.

The sg3_utils package and some more complex utilities have html pages: sg3_utils: https://sg.danny.cz/sg/sg3_utils.html sg_ses: https://sg.danny.cz/sg/sg_ses.html sg_dd: https://sg.danny.cz/sg/sg_dd.html

A tarball (and zip) of all the manpages from the previous release are here: https://sg.danny.cz/sg/p/sg3_utils_man_html.tgz https://sg.danny.cz/sg/p/sg3_utils_man_html.zip

There is a html rendering of the sg3_utils manpage in the same directory as this README file called sg3_utils.man8.html .

The previous README file is now called README.details plus there are these OS specific files: README.freebsd , README.solaris , README.tru64 and README.win32 . To know the current state of the package the ChangeLog file is the good reference.

The author's primary source code repository uses subversion and is on the author's equipment (a RPi). One advantage of subversion is its revision numbers which are simply integers starting at 1 and ascending. For this package the current revision is 928 . The subversion repository is mirrored in git (using "git svn" tools) here: https://github.com/doug-gilbert/sg3_utils

Douglas Gilbert 26th April 2023