Public Member Functions | Public Attributes | Private Member Functions | Private Attributes

fio_uvListMgr Class Reference
[Classes for large file i/o, relative file, and DataBase]

#include <fio_uvListMgr.h>

Collaboration diagram for fio_uvListMgr:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 fio_uvListMgr ()
 fio_uvListMgr (const fio_uvListMgr &a_listMgr)
 ~fio_uvListMgr ()
fio_uvListMgroperator= (const fio_uvListMgr &a_listMgr)
int GetRecommendBufferSize ()
void SetRecommendBufferSize (int a_size)
 Set recommended buffer size, Tuning reason.
void Init (fio_RelFileTrans *a_relFileTrans_p, fio_uvListHeader_t *a_uvListHeader_p, UInt16_t a_vectorSize, ivd_FilePosition_t a_headerPosition, int a_recommendUVBufferSize)
 initialization of instance
void RemoveVectorFromBuff (fio_RecordIDX_t a_idx)
 When Replayed transaction is in progress remove all used vector from UV buffer before DBupdate.
void AddVectorToBuff (fio_RecordIDX_t a_idx)
 When Replayed transaciton is in progres add all relesed vector to UV buffer after DBupdate.
ivd_RecordIDX_t GetVectorIDX ()
 Return 0 if no UV exist othervice return index of vector.
void GetBunchOfVectorIDXes (ivd_RecIdxVec_t &a_recIdxVec, ivd_RecIdxVec_i &a_recIdxVectIter, fio_Transaction *a_transaction_p)
void PutVector (fio_RecordIDX_t a_idx)
 Put unused vector to list.
void Synchronize (fio_Transaction *a_transaction_p)
 To have consistent DB sinhronize method, flush all unused record form list to disk.

Public Attributes

 log_CLASSID_m

Private Member Functions

void WriteUV (unsigned int a_indeces, fio_Transaction *a_transaction_p)
 When list grow more than HWM, then Write must flush a_indeces to disk.
void ReadUV ()
 When list shrink to low WM then Read some number of UV form disk.
bool IsElemInBuffer (fio_RecordIDX_t a_newIdx)
 check if adding element is already in buffer, then throw internal error.

Private Attributes

fio_RelFileTransm_relFileTrans_p
 pointer to its relative file, don't need to destruct
fio_uvListHeader_tm_uvListHeader_p
 pointer to particular record in LoLoUV
UInt16_t m_vectorSize
 the size of vectore that are in list (used for vector index in list)
ivd_FilePosition_t m_headerPosition
 header file position
fio_RecordIDX_v_t m_recordIDX_v
 vector of indeces that point to unused vecto
cmn_Mutex m_uvListMgr_x
 a mutex to concurentli access to list mgr members of indices;
UInt32_t m_highWaterMark
 When number of indeces reach HWM, then flush (write) UVrecords to disk.
UInt32_t m_middWaterMark
 the read or write proces is stopped when middle WM is reach.

Detailed Description

Definition at line 53 of file fio_uvListMgr.h.


Constructor & Destructor Documentation

fio_uvListMgr::fio_uvListMgr (  ) 

Definition at line 54 of file fio_uvListMgr.cpp.

fio_uvListMgr::fio_uvListMgr ( const fio_uvListMgr a_listMgr  )  [inline]

Definition at line 57 of file fio_uvListMgr.h.

{};

fio_uvListMgr::~fio_uvListMgr (  ) 

Definition at line 67 of file fio_uvListMgr.cpp.

                              {
//     log_FUNC_m(~fio_uvListMgr);
    //Empty
}


Member Function Documentation

void fio_uvListMgr::AddVectorToBuff ( fio_RecordIDX_t  a_idx  ) 

When Replayed transaciton is in progres add all relesed vector to UV buffer after DBupdate.

Definition at line 115 of file fio_uvListMgr.cpp.

References fio_File::GetFileID(), IsElemInBuffer(), log_ERR_m, log_FUNC_m, m_recordIDX_v, m_relFileTrans_p, and m_vectorSize.

                                                         {

// DO NOT LOCK    cmn_MutexLock l(m_uvListMgr_x);
    if (a_idx == 0) {
        log_FUNC_m(AddVectorToBuff);
        log_ERR_m("UV to mem buff. Idx = 0  RelFileID = " << (int)m_relFileTrans_p->GetFileID()
               << "  vectorSize = " << m_vectorSize);        
    }
//    log_DBG_m(dbg_DETAIL, "AddVec : " << a_idx);
    if (IsElemInBuffer(a_idx)) {
        return;
    }
    m_recordIDX_v.push_back(a_idx);
}

Here is the call graph for this function:

void fio_uvListMgr::GetBunchOfVectorIDXes ( ivd_RecIdxVec_t &  a_recIdxVec,
ivd_RecIdxVec_i &  a_recIdxVectIter,
fio_Transaction a_transaction_p 
)

Definition at line 159 of file fio_uvListMgr.cpp.

References cmn_Num2Str(), fio_File::GetFileID(), fio_File::GetFileName(), GetRecommendBufferSize(), ie_UV_NOT_IN_BUFFER, log_FUNC_m, m_recordIDX_v, m_relFileTrans_p, m_uvListMgr_x, m_vectorSize, ReadUV(), relFileID, SetRecommendBufferSize(), and fio_Transaction::VectorIsUsed().

                                                  {

    cmn_MutexLock l(m_uvListMgr_x);

    int uvBuffSize = 0;
    if (m_recordIDX_v.size() < a_recIdxVec.size()) {
        uvBuffSize = GetRecommendBufferSize();
        SetRecommendBufferSize(a_recIdxVec.size()); // enought space for all vector
        ReadUV();                    // read some UV from file mini transaction
    }

    if (m_recordIDX_v.size() < a_recIdxVec.size()) { // not enought free vectors
// TODO check content of this throw
        log_FUNC_m(GetBunchOfVectorIDXes);
        throw ivd_InternalError(ie_UV_NOT_IN_BUFFER, 
            "File: " + m_relFileTrans_p->GetFileName() + 
             ", vectore of size " + cmn_Num2Str(m_vectorSize) + " not enought found.", true);
        return;
    }

    UInt16_t     vectSize = m_vectorSize;
    ivd_RelFileID_t relFileID   = m_relFileTrans_p->GetFileID();
    while (  (m_recordIDX_v.size() > 0)
          && (a_recIdxVectIter != a_recIdxVec.end())) {

        fio_RecordIDX_t idx = m_recordIDX_v.back();
        m_recordIDX_v.pop_back();
        *a_recIdxVectIter++   = idx;
        a_transaction_p->VectorIsUsed(relFileID, vectSize, idx);
    }

    // return back UV buffer size to previous value 
    if (uvBuffSize > 0) {
        SetRecommendBufferSize(uvBuffSize);
    }
}

Here is the call graph for this function:

int fio_uvListMgr::GetRecommendBufferSize (  )  [inline]

Definition at line 101 of file fio_uvListMgr.h.

Referenced by GetBunchOfVectorIDXes().

{ return m_middWaterMark;};

Here is the caller graph for this function:

fio_RecordIDX_t fio_uvListMgr::GetVectorIDX (  ) 

Return 0 if no UV exist othervice return index of vector.

Definition at line 132 of file fio_uvListMgr.cpp.

References cmn_Num2Str(), fio_File::GetFileName(), ie_UV_NOT_IN_BUFFER, log_FUNC_m, m_recordIDX_v, m_relFileTrans_p, m_uvListMgr_x, m_vectorSize, and ReadUV().

                                            {
        
    cmn_MutexLock l(m_uvListMgr_x);

    if (m_recordIDX_v.size() == 0) { // in future m_lowWM could be here;
        ReadUV();                    // read some UV from file mini transaction

        if (m_recordIDX_v.size() == 0) { // if still nothing to get
            log_FUNC_m(GetVectorIDX);
    // TODO check content of this throw
            throw ivd_InternalError(ie_UV_NOT_IN_BUFFER, 
                "File: " + m_relFileTrans_p->GetFileName() + 
                 ", vector of size " + cmn_Num2Str(m_vectorSize) + " not found.", true);
            return 0;
        }
    }
//    log_DBG_m(dbg_DETAIL, m_recordIDX_v.size() << " free vectors in buffer.");

    // get last element from list
    fio_RecordIDX_t idx = m_recordIDX_v.back();
    m_recordIDX_v.pop_back();

    return idx;
}

Here is the call graph for this function:

void fio_uvListMgr::Init ( fio_RelFileTrans a_relFileTrans_p,
fio_uvListHeader_t a_uvListHeader_p,
UInt16_t  a_vectorSize,
ivd_FilePosition_t  a_headerPosition,
int  a_recommendUVBufferSize 
)

initialization of instance

Definition at line 74 of file fio_uvListMgr.cpp.

References m_headerPosition, m_relFileTrans_p, m_uvListHeader_p, m_vectorSize, and SetRecommendBufferSize().

                                                      {
        
    m_relFileTrans_p = a_relFileTrans_p;
    m_uvListHeader_p = a_uvListHeader_p;
    m_vectorSize     = a_vectorSize; // size of vector
    m_headerPosition = a_headerPosition;

    SetRecommendBufferSize(a_recommendUVBufferSize);
}

Here is the call graph for this function:

bool fio_uvListMgr::IsElemInBuffer ( fio_RecordIDX_t  a_newIdx  )  [private]

check if adding element is already in buffer, then throw internal error.

method is called before push_back()

Definition at line 405 of file fio_uvListMgr.cpp.

References evt_CRITICAL, fio_File::GetFileName(), log_FUNC_m, log_WriteEvent(), m_recordIDX_v, m_relFileTrans_p, and m_vectorSize.

Referenced by AddVectorToBuff(), and PutVector().

                                                           {

    fio_RecordIDX_t *ip = &*m_recordIDX_v.begin();
    fio_RecordIDX_t *ep = &*m_recordIDX_v.end();
    for (; ip < ep; ++ip) {
        if (*ip == a_newIdx) {
            log_FUNC_m(IsElemInBuffer);
            ostringstream sstr;
            sstr << "File: " << m_relFileTrans_p->GetFileName() 
                 << " vector at index " << a_newIdx 
                 << " of size " << m_vectorSize 
                 << " was released twice." << endl
                 << " A DB recovery needs to be done in near future."
                 << " Pack DB and contact support for further steps.";
            log_WriteEvent(evt_CRITICAL, sstr.str());
//            throw ivd_InternalError(ie_UV_ALREADY_IN_BUFFER, sstr.str(), true);
            return true;
        }
    }
    return false;
}

Here is the call graph for this function:

Here is the caller graph for this function:

fio_uvListMgr& fio_uvListMgr::operator= ( const fio_uvListMgr a_listMgr  )  [inline]

Definition at line 99 of file fio_uvListMgr.h.

{ return *this; };

void fio_uvListMgr::PutVector ( fio_RecordIDX_t  a_idx  ) 

Put unused vector to list.

Definition at line 202 of file fio_uvListMgr.cpp.

References fio_Transaction::EndTransaction(), fio_Transaction::ExclusiveStartTransaction(), fio_DataBase::GetTransObj(), IsElemInBuffer(), fio_RelFileTrans::m_database_p, m_highWaterMark, m_middWaterMark, m_recordIDX_v, m_relFileTrans_p, m_uvListMgr_x, fio_DataBase::ReleaseTransObj(), and WriteUV().

                                                   {
//    log_FUNC_m(PutVector);

    cmn_MutexLock l(m_uvListMgr_x);
    if (IsElemInBuffer(a_idx)) {
        return;
    }
    m_recordIDX_v.push_back(a_idx);

    if (m_recordIDX_v.size() >= m_highWaterMark) { 
        // get transaction object to adopt DB access
        fio_Transaction *miniTrans_p = m_relFileTrans_p->m_database_p->GetTransObj();
        // FIX bug 634
        miniTrans_p->ExclusiveStartTransaction();
        // end of fix

        WriteUV(m_middWaterMark, miniTrans_p); // Write some UV to file;

        miniTrans_p->EndTransaction();
        // release transaction object as not needed
        m_relFileTrans_p->m_database_p->ReleaseTransObj(&miniTrans_p);
    }
}

Here is the call graph for this function:

void fio_uvListMgr::ReadUV (  )  [private]

When list shrink to low WM then Read some number of UV form disk.

The fact is that all bind vectors in file are declarated as used.

Definition at line 298 of file fio_uvListMgr.cpp.

References dbg_DETAIL, cmn_Global::dirs, fio_Transaction::EndTransaction(), fio_Transaction::ExclusiveStartTransaction(), fio_uvListHeader_t::firstUnusedIDX, g_cmn, ivd_BaseException::GetError(), fio_File::GetFileID(), fio_RelFile::GetFirstNewIDX(), fio_DataBase::GetTransObj(), fio_RelFile::Idx2Pos(), log_DBG_m, log_FUNC_m, log_WRN_m, fio_RelFileTrans::m_database_p, m_headerPosition, fio_UV_t::m_idx, m_middWaterMark, ivd_Product::m_nameShort, m_recordIDX_v, m_relFileTrans_p, m_uvListHeader_p, m_vectorSize, fio_uvListHeader_t::numOfElements, ivd_Directories::part, fio_RelFileTrans::PosWrite(), cmn_Global::prod, fio_RelFile::ReadNextUnusedVectorIDX(), fio_DataBase::ReleaseTransObj(), relFileID, fio_Transaction::SetMiniTransaction(), and fio_Transaction::VectorIsFreed().

Referenced by GetBunchOfVectorIDXes(), and GetVectorIDX().

                           {
    log_FUNC_m(ReadUV);
    // read as much uv to count m_middWaterMark

    // prepare a mini transaciton
    fio_Transaction *a_miniTrans_p;
    // get transaction object to adopt DB access
    a_miniTrans_p = m_relFileTrans_p->m_database_p->GetTransObj();
    a_miniTrans_p->SetMiniTransaction(true);
    // FIX bug 634
    a_miniTrans_p->ExclusiveStartTransaction();
    // end of fix

    UInt16_t     vecSize = (m_vectorSize);
    ivd_RelFileID_t relFileID  = m_relFileTrans_p->GetFileID();

    unsigned int       i =  m_middWaterMark - m_recordIDX_v.size();
    unsigned int  toRead = i;
    //log_DBG_m(dbg_DETAIL, "Num of needed vector " << i 
    //                << " first UnusedIdx " << uv.m_idx);
    fio_UV_t uv(relFileID, vecSize, m_uvListHeader_p->firstUnusedIDX);
    ivd_RecordIDX_t nextIDX ;
    while (  (i    > 0)
          && (uv.m_idx != 0)) {

        try {
            nextIDX = m_relFileTrans_p->ReadNextUnusedVectorIDX(uv.m_idx, vecSize - 1);
        }
        catch (ivd_Error &err) {
            if (err.GetError() == ie_INV_UNUSED_VECT_STRUCT) {
                log_WRN_m("Corruption in a journal file was detected. It could "
                       << "be caused by hardware problem or shortage of disk "
                       << "space in " << g_cmn.dirs.part.c_str()
                       << " directory. " << g_cmn.prod.m_nameShort.c_str()
                       << " has performed correction. Please run hsmfs "
                       << "check soon to detect/correct potential "
                       << "inconsistencies in the HSMDB and/or FSCDB.");
                break;
            }
            throw;
        }
        // put note to transaction that this idx will be "free" not bind in file
        a_miniTrans_p->VectorIsFreed(uv);
        uv.m_idx = nextIDX;
        log_DBG_m(dbg_DETAIL, "UnusedIdx " << uv.m_idx);
        i--;
    }


    if (i != toRead) { // if file header is change than update it
        //log_DBG_m(dbg_DETAIL, "Update header: previous unusedIDX " 
        //                      << m_uvListHeader_p->firstUnusedIDX
        //                      << "  previous numbef of element " 
        //                      << m_uvListHeader_p->numOfElements);
        m_uvListHeader_p->firstUnusedIDX = uv.m_idx; // new first UV
        m_uvListHeader_p->numOfElements -= (toRead - i); // number of UV
        //log_DBG_m(dbg_DETAIL, "new idx = "             << idx
        //                      << "  num of used vectors " << (toRead - i)
        //                      << "  new num of elements " << m_uvListHeader_p->numOfElements);

        fio_uvListHeader_t uvlHeader(*m_uvListHeader_p);

// ALTERNATIVE call m_relFileTrans_p->UpdateListOfUnusedVector(m_vectorSize, m_transaction_p);
// instead of PosWrite,  not used now, but to have in mind.
        m_relFileTrans_p->PosWrite(m_headerPosition, // write changes
                                   &uvlHeader, 
                                   sizeof(fio_uvListHeader_t), 
                                   a_miniTrans_p);
    }
    
    // if no more UV in file then get it form the end of the file
    if (i > 0) {
        // all ideces at once
        int numOfIdx = i * vecSize;
        // idxIter point to first new
        uv.m_idx = m_relFileTrans_p->GetFirstNewIDX(numOfIdx) + numOfIdx;
        // write last byte to file to increase its size, 
        // because this space is now in evidence
        static char zerroByte = 0;
        m_relFileTrans_p->PosWrite(m_relFileTrans_p->Idx2Pos(uv.m_idx) - 1, 
                                  &zerroByte, 
                                  1, 
                                  a_miniTrans_p);
        //log_DBG_m(dbg_DETAIL, "Get new vectors from the end of the file " 
        //                    << m_relFileTrans_p->GetFileName()
        //                    << " first free idx " << uv.m_idx);
        // write to vector in reverse order, because index is got from back and 
        // on this way are got on increase sequence
        while (i > 0) {
            uv.m_idx -= vecSize; // next free idx
            a_miniTrans_p->VectorIsFreed(uv);
            i--;
        }
    }
    // end transaction will fill up the UV buffer;
    a_miniTrans_p->EndTransaction(); 
    a_miniTrans_p->SetMiniTransaction(false);
    // release transaction object as not needed
    m_relFileTrans_p->m_database_p->ReleaseTransObj(&a_miniTrans_p);
}

Here is the call graph for this function:

Here is the caller graph for this function:

void fio_uvListMgr::RemoveVectorFromBuff ( fio_RecordIDX_t  a_idx  ) 

When Replayed transaction is in progress remove all used vector from UV buffer before DBupdate.

Definition at line 91 of file fio_uvListMgr.cpp.

References fio_File::GetFileName(), ie_UV_NOT_IN_BUFFER, log_FUNC_m, m_recordIDX_v, m_relFileTrans_p, and m_vectorSize.

                                                              {

//DO NOT LOCK     cmn_MutexLock l(m_uvListMgr_x);
    fio_RecordIDX_t *ip = &*m_recordIDX_v.begin();
    fio_RecordIDX_t *ep = &*m_recordIDX_v.end();
    for (; ip < ep; ++ip) {
        if (*ip == a_idx) {
            *ip = *--ep; // overwrite element with last one
            m_recordIDX_v.pop_back(); // remove last            
            return;
        }
    }
    log_FUNC_m(RemoveVectorFromBuff);
    char size_s[20];
    char idx_s[20];
    sprintf(size_s, "%d", m_vectorSize);
    sprintf(idx_s, "%d", a_idx);
    throw ivd_InternalError(ie_UV_NOT_IN_BUFFER, 
        "File: " + m_relFileTrans_p->GetFileName() + 
        " Index " + idx_s + " of " + size_s + " element vector missing.", true);
}

Here is the call graph for this function:

void fio_uvListMgr::SetRecommendBufferSize ( int  a_size  )  [inline]

Set recommended buffer size, Tuning reason.

Definition at line 104 of file fio_uvListMgr.h.

Referenced by GetBunchOfVectorIDXes(), and Init().

                                                   {
        m_highWaterMark = a_size * 2;
        m_middWaterMark = a_size;
    }

Here is the caller graph for this function:

void fio_uvListMgr::Synchronize ( fio_Transaction a_transaction_p  ) 

To have consistent DB sinhronize method, flush all unused record form list to disk.

used by DBLock method;

Definition at line 230 of file fio_uvListMgr.cpp.

References m_recordIDX_v, m_uvListMgr_x, and WriteUV().

                                                                 {

    cmn_MutexLock l(m_uvListMgr_x);
    if (m_recordIDX_v.size() > 0) {
        WriteUV(m_recordIDX_v.size(), a_transaction_p);
    }
}

Here is the call graph for this function:

void fio_uvListMgr::WriteUV ( unsigned int  a_indeces,
fio_Transaction a_transaction_p 
) [private]

When list grow more than HWM, then Write must flush a_indeces to disk.

Definition at line 243 of file fio_uvListMgr.cpp.

References fio_UNUSED_REC_CLEAR_BYTES_d, fio_uvListHeader_t::firstUnusedIDX, fio_File::GetFileID(), fio_RelFile::GetRecordSize(), fio_RelFile::Idx2Pos(), m_headerPosition, fio_UV_t::m_idx, m_recordIDX_v, m_relFileTrans_p, m_uvListHeader_p, m_vectorSize, fio_uvListHeader_t::numOfElements, fio_RelFileTrans::PosWrite(), relFileID, and fio_Transaction::VectorIsUsed().

Referenced by PutVector(), and Synchronize().

                                                   {

//    log_FUNC_m(WriteUV);

    UInt16_t     vecSize = (m_vectorSize);
    ivd_RelFileID_t relFileID  = m_relFileTrans_p->GetFileID();

//    fio_RecordIDX_t idx = 0;

    fio_UV_t uv(relFileID, vecSize, 0);
    unsigned int i = a_indeces;
    while (  (i > 0)
          && (m_recordIDX_v.size() > 0)) {

        uv.m_idx = m_recordIDX_v.back();
        m_recordIDX_v.pop_back();

        fio_UnusedRec_t unusedRec(m_uvListHeader_p->firstUnusedIDX, vecSize - 1);

        // vector is removed from buffer, so is used now
        a_miniTrans_p->VectorIsUsed(uv);

        // released record point to current in list
        ivd_FileBufSize_t recSize = m_relFileTrans_p->GetRecordSize();
        m_relFileTrans_p->PosWrite(m_relFileTrans_p->Idx2Pos(uv.m_idx), 
                                  &unusedRec, 
                                  recSize > fio_UNUSED_REC_CLEAR_BYTES_d
                                  ? fio_UNUSED_REC_CLEAR_BYTES_d
                                  : recSize, 
                                  a_miniTrans_p);

        m_uvListHeader_p->firstUnusedIDX = uv.m_idx;  // header now point to it
        m_uvListHeader_p->numOfElements++;            // increment number of UV

        i--;
    }

    if (i != a_indeces) {  // some UV were written, header update
        fio_uvListHeader_t uvlHeader(*m_uvListHeader_p);

        m_relFileTrans_p->PosWrite(m_headerPosition, // write changes
                                   &uvlHeader, 
                                   sizeof(fio_uvListHeader_t), 
                                   a_miniTrans_p);
    }
}

Here is the call graph for this function:

Here is the caller graph for this function:


Member Data Documentation

Definition at line 95 of file fio_uvListMgr.h.

header file position

Definition at line 79 of file fio_uvListMgr.h.

Referenced by Init(), ReadUV(), and WriteUV().

When number of indeces reach HWM, then flush (write) UVrecords to disk.

Definition at line 88 of file fio_uvListMgr.h.

Referenced by PutVector().

the read or write proces is stopped when middle WM is reach.

Definition at line 90 of file fio_uvListMgr.h.

Referenced by PutVector(), and ReadUV().

vector of indeces that point to unused vecto

Definition at line 82 of file fio_uvListMgr.h.

Referenced by AddVectorToBuff(), GetBunchOfVectorIDXes(), GetVectorIDX(), IsElemInBuffer(), PutVector(), ReadUV(), RemoveVectorFromBuff(), Synchronize(), and WriteUV().

pointer to its relative file, don't need to destruct

Definition at line 70 of file fio_uvListMgr.h.

Referenced by AddVectorToBuff(), GetBunchOfVectorIDXes(), GetVectorIDX(), Init(), IsElemInBuffer(), PutVector(), ReadUV(), RemoveVectorFromBuff(), and WriteUV().

pointer to particular record in LoLoUV

Definition at line 73 of file fio_uvListMgr.h.

Referenced by Init(), ReadUV(), and WriteUV().

a mutex to concurentli access to list mgr members of indices;

Definition at line 85 of file fio_uvListMgr.h.

Referenced by GetBunchOfVectorIDXes(), GetVectorIDX(), PutVector(), and Synchronize().

the size of vectore that are in list (used for vector index in list)

Definition at line 76 of file fio_uvListMgr.h.

Referenced by AddVectorToBuff(), GetBunchOfVectorIDXes(), GetVectorIDX(), Init(), IsElemInBuffer(), ReadUV(), RemoveVectorFromBuff(), and WriteUV().


The documentation for this class was generated from the following files: