Description of the internal data format

See also IVD Data format. for API documentation reference.

Introduction

IVD Data format consists of:

• Media partition layout
• Fixed-size blocks, variable-length records and data packaging

Blocks are low level components that is used for writing to the tape: one block is written with one SCSI write command. Blocks can contain records or can be raw data blocks.

Records are used to pack the meta-data and file contents into the blocks.

Layout of media volumes on a medium

Layout of blocks on one medium volume and layout of records inside the data block:

Medium volume layout consists of:

• ECMA/ANSI compliant medium volume header (df_ECMAVolumeLabel_t), which is followed by one empty data block for future enhancements and a file mark. This blocks are written during medium initialization
• Data blocks are written after medium volume header. Each of the blocks (except the RAW data blocks) has a block header (df_BlockHeader_t)
• A file mark is written to tape when medium volume is almost full and then
• FRI blocks are copied from temporary location on disk to the media volume.

Structure of a medium volume and records inside the block

IVD Data format is used for:

• writing data and Fast Recovery Information (FRI) to the tape
• creating temporary disk buffer files for migration
• gathering FRI on disk for non-closed media volumes

Description of the layout

Structure of all records inside the data format is the same in all cases.

Blocks and records must be written in appropriate order.

df_BlockHeader_t contains all of the information that is written to each of non-raw blocks.

Blocks that were written in one migration are numbered starting from 1 including raw data blocks. Each migration restarts the numbering. However, if the block numbers are not consistent, it is an indication of corrupted data format.

Block cookie is another measure of detecting data format corruption. Block magic cookie must be equal to df_blkCookie_c.

Each block can be one of the following types:

• blk_Data_c for data blocks
• blk_FRI_c for FRI records

Blocks of different types can be mixed together. They must be in separate files or separated by file marks on tape.

All records inside the blocks have common df_RecCmn_t part that contains basic record information: sequence number, magic cookie, record type, etc.

Records that are written in one migration are also numbered, starting from 1. Numbering is not reset with each new block, but it continues. It is reset with each migration.

IVD records also have magic cookie. It must be equal to df_recCookie_c.

Records were designed to allow flexibility, extensibility and performance.

Types of records that are used to package information are:

• Volume Header (df_RecCmn_t + df_RecVolHdr_t)
• File Header (df_RecCmn_t + df_RecFile_t)
• End of file (df_RecCmn_t + df_RecFileEnd_t)
• Byte Stream Start (df_RecCmn_t + df_RecByteStreamStart_t)
• Byte Stream End (df_RecCmn_t + df_RecByteStreamEnd_t)
• Embedded Data (df_RecCmn_t + df_RecEmbeddedData_t)
• FRI Start (df_RecCmn_t + df_RecFRIStart_t)
• FRI End (df_RecCmn_t + df_RecFRIEnd_t)
• Split Info (df_RecCmn_t + df_RecFRISpInfo_t)

Correct sequence of records can be seen on the following state diagram:

State diagram to read valid data format structure.