Public Member Functions | Static Private Member Functions | Private Attributes

fsc_CollectorRedunCopy Class Reference
[FSC]

#include <fsc_CollectorRedunCopy.h>

Inheritance diagram for fsc_CollectorRedunCopy:
Inheritance graph
[legend]
Collaboration diagram for fsc_CollectorRedunCopy:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 fsc_CollectorRedunCopy (fsc_RedunCopyResult_v &a_results, vector< UInt32_t > &a_unavailFilesPerVol, fsc_MediumStatus_v &a_medStat, ivd_MediaKey_t a_origMedKey, vector< UInt32_t > a_origMedVol, bool a_best_effort)
virtual ~fsc_CollectorRedunCopy (void)
virtual bool ProcRecGeneration (fsc_Generation_t &a_gen, UInt32_t a_status)
 Called when new file in process.
virtual bool PostRecCopy (fsc_Copy_t &a_copy)
 Called after each copy processed.
virtual bool ProcRecSplit (fsc_Split_t &a_split)
 Called when generation or copy has splits.
virtual bool ProcRecMedPosMgr (fsc_MediaPosition_t &a_medPos)
 Called when media position is known.
virtual bool PostCopiesProc ()
 Called when all copies are preocessed.

Static Private Member Functions

static UInt64_t KeyGen (ivd_MediaKey_t a_mediaKey, ivd_MedVolNum_t a_medVolNum)
 Key to searc element in stx_cache.

Private Attributes

 log_CLASSID_m
fsc_RedunCopyResult_vm_results
const fsc_MediumStatus_v m_medStat
const ivd_MediaKey_t m_origMedKey
vector< UInt32_tm_origMedVol
UInt32_t m_currentVolume
vector< UInt32_t > & m_unavailFilesPerVol
const Int32_t m_origPoolID
const cmn_Path m_redunCopyDir
stx_Cache m_cache
 MedVolume file storage.
UInt32_t m_numOfRedundCopies
UInt32_t m_numOfUnavaillCopies
UInt32_t m_numOfPurgedCopies
UInt32_t m_numOfSkippedCopies
ivd_MigrationID_t m_migID
ivd_FileSize_t m_streamSize
 stream size stored in generation
UInt8_t m_dataType
 each generation set this member
ivd_FileSize_t m_dataOffset
 generation and MedPos set to 0, split set its value
ivd_FileSize_t m_dataSize
 generation and MedPos set to m_streamSize, split set its value
ivd_CopiesPos_v_t m_cpyMedPos_v
 each generation clear and fill its media position
ivd_CopiesPos_v_i m_copyIter
ivd_MediaPos_v_i m_medPosIter
bool m_best_effort
 do redundant copy collection on best effort base
bool m_doRedunCopy
 Set if copy exists on original medium, reset for each gen.

Detailed Description

Definition at line 46 of file fsc_CollectorRedunCopy.h.


Constructor & Destructor Documentation

fsc_CollectorRedunCopy::fsc_CollectorRedunCopy ( fsc_RedunCopyResult_v a_results,
vector< UInt32_t > &  a_unavailFilesPerVol,
fsc_MediumStatus_v a_medStat,
ivd_MediaKey_t  a_origMedKey,
vector< UInt32_t a_origMedVol,
bool  a_best_effort 
)

Definition at line 56 of file fsc_CollectorRedunCopy.cpp.

References cmn_CleanDir(), dbg_DETAIL, log_DBG_m, log_FUNC_m, and m_redunCopyDir.

        :
        m_results(a_results),
        m_medStat(a_medStat),
        m_origMedKey(a_origMedKey),
        m_origMedVol(a_origMedVol),
        m_currentVolume(0),
        m_unavailFilesPerVol(a_unavailFilesPerVol),
        m_origPoolID(a_medStat[a_origMedKey].mediaPoolKey),
        m_redunCopyDir(fsc_GetRedunCopyDir(a_origMedKey)),
        m_cache(1024), // small amount of different media is possible
        m_numOfRedundCopies(0),
        m_numOfUnavaillCopies(0),
        m_numOfSkippedCopies(0),
        m_numOfPurgedCopies(0),
        m_best_effort(a_best_effort)
{
    log_FUNC_m(fsc_CollectorRedunCopy);
    log_DBG_m(dbg_DETAIL, "Create dir " << m_redunCopyDir);
    cmn_CleanDir(m_redunCopyDir);
}

Here is the call graph for this function:

fsc_CollectorRedunCopy::~fsc_CollectorRedunCopy ( void   )  [virtual]

Definition at line 85 of file fsc_CollectorRedunCopy.cpp.

References log_WriteEvent(), m_best_effort, m_numOfPurgedCopies, m_numOfRedundCopies, m_numOfSkippedCopies, m_numOfUnavaillCopies, and m_origMedKey.

                                                {


    ostringstream sstr;
    sstr << "Finished FSC redundant copy collect for medium : " << m_origMedKey
         << ", Collected = " << m_numOfRedundCopies
         << ", Skipped (already copied) = " << m_numOfSkippedCopies;
    if( m_best_effort){
        sstr << ", Purged (on best effort base) = " << m_numOfPurgedCopies;
    }
    sstr << ", WRN Unavailable = " << m_numOfUnavaillCopies;
    log_WriteEvent(sstr.str(), "REDUNDANT COPY");
}

Here is the call graph for this function:


Member Function Documentation

static UInt64_t fsc_CollectorRedunCopy::KeyGen ( ivd_MediaKey_t  a_mediaKey,
ivd_MedVolNum_t  a_medVolNum 
) [inline, static, private]

Key to searc element in stx_cache.

Definition at line 83 of file fsc_CollectorRedunCopy.h.

Referenced by PostCopiesProc().

                                                    {
        return UInt64_t(a_mediaKey << 6) + a_medVolNum;
    };

Here is the caller graph for this function:

bool fsc_CollectorRedunCopy::PostCopiesProc (  )  [virtual]

Called when all copies are preocessed.

Find best copy.

Scan through all copy. Check if copy is useful. Copy is useful if has all splits and all media are usable. Copy is better if lays on faster medium or contains less splits. Count all copies on origin pool. Allow only one redundant copy per pool.

Reimplemented from fsc_RawCollector.

Definition at line 178 of file fsc_CollectorRedunCopy.cpp.

References ivd_FileLocationData_t::blockOffset, ivd_MediaPos_t::blockOffset, dbg_DETAIL, ivd_FileLocationData_t::fileID, stx_Cache::Get(), stx_Cache::Insert(), KeyGen(), log_DBG_m, log_FUNC_m, log_WriteEvent(), log_WRN_m, m_best_effort, m_cache, m_cpyMedPos_v, fsc_RawCollector::m_curFileID, m_currentVolume, m_doRedunCopy, m_medStat, m_migID, m_numOfPurgedCopies, m_numOfRedundCopies, m_numOfSkippedCopies, m_numOfUnavaillCopies, m_origMedKey, m_origPoolID, m_results, m_streamSize, m_unavailFilesPerVol, ivd_MediaPos_t::mediaKey, ivd_MediaPos_t::medVolNum, ivd_FileLocationData_t::migrationID, mt_DISK, NULL, rmdb_MEDIUM_OFFLINE, rmdb_MEDIUM_UNUSABLE, ivd_MediaPos_t::splitOffset, ivd_FileLocationData_t::splitSize, ivd_MediaPos_t::splitSize, and fsc_RedunCopyDataVolStorage::WriteToBuffer().

                                            {
    log_FUNC_m(PostCopiesProc);

    if (m_doRedunCopy) {
        Int32_t goodCopiesPerPool = 0;
        Int32_t   bestCopyMedType = 0;
        Int32_t  bestCopySplitCnt = 0;
        log_DBG_m(dbg_DETAIL, m_cpyMedPos_v.size() << " generation's copies foud." );
        ivd_CopiesPos_v_i bestCopyIter = m_cpyMedPos_v.end();
        ivd_CopiesPos_v_i     copyIter = m_cpyMedPos_v.begin();
        for (; copyIter != m_cpyMedPos_v.end(); ++copyIter) {
            // check if contain all splits
            ivd_MediaPos_v_i mpIter = copyIter->begin();

            ivd_MediaKey_t medKey   = mpIter->mediaKey;
            if (medKey == m_origMedKey) {
                //this is the original copy which we required copyContent
                m_currentVolume = mpIter->medVolNum;
                log_DBG_m(dbg_DETAIL, "m_currentVolume " << m_currentVolume);
            }

            Int32_t  mediaPoolKey   = m_medStat[medKey].mediaPoolKey;
            Int32_t  slowestMedType = m_medStat[medKey].medType;
            Int64_t    offset = 0;
            log_DBG_m(dbg_DETAIL, "medKey " << medKey);
            log_DBG_m(dbg_DETAIL, "mediaPoolKey " << mediaPoolKey);
            log_DBG_m(dbg_DETAIL, "slowestMedType " << slowestMedType);
            log_DBG_m(dbg_DETAIL, "m_streamSize " << m_streamSize );

            log_DBG_m(dbg_DETAIL, copyIter->size() << " split's found." );
            ivd_MediaPos_v_i  mpEnd = copyIter->end();
            for (; mpIter != mpEnd; ++mpIter) {
                Int32_t medStatus = m_medStat[mpIter->mediaKey].status;
                log_DBG_m(dbg_DETAIL, "offset == mpIter->splitOffset "
                                    << offset << " == " << mpIter->splitOffset
                                    << ", medStatus " << medStatus
                                    << ", rmdb_MEDIUM_UNUSABLE = " << rmdb_MEDIUM_UNUSABLE
                                    << ", rmdb_MEDIUM_OFFLINE = " << rmdb_MEDIUM_OFFLINE);
                if (   (offset == mpIter->splitOffset)
                    && ((medStatus & rmdb_MEDIUM_UNUSABLE) == 0)
                    && ((medStatus & rmdb_MEDIUM_OFFLINE) == 0)) {
                    offset += mpIter->splitSize;
                    Int32_t medType = m_medStat[mpIter->mediaKey].medType;
                    log_DBG_m(dbg_DETAIL, "slowestMedType " << slowestMedType
                                        << ", medType " << medType
                                        << ", mt_DISK " << mt_DISK);
                    if (  slowestMedType == 0
                        || medType != mt_DISK) {
                        slowestMedType = m_medStat[mpIter->mediaKey].medType;
                    }
                }
                else { //
                    log_DBG_m(dbg_DETAIL, "Next copy.")
                    goto nextCopy;
                }
            }

            log_DBG_m(dbg_DETAIL, "m_streamSize == offset " <<
                                m_streamSize << " == " << offset );
            if (m_streamSize == offset) {
                log_DBG_m(dbg_DETAIL, "mediaPoolKey == m_origPoolID "
                                    << mediaPoolKey << " == " << m_origPoolID);
//                                    << ", copyFromOrigMed " << copyFromOrigMed);
                if (  mediaPoolKey == m_origPoolID) {
                    ++goodCopiesPerPool;
                }
                log_DBG_m(dbg_DETAIL, "goodCopiesPerPool "  << goodCopiesPerPool);
                log_DBG_m(dbg_DETAIL, "bestCopyIter == m_cpyMedPos_v.end() " << boolalpha << (bestCopyIter == m_cpyMedPos_v.end())
                               << ", bestCopyMedType < slowestMedType "
                               << bestCopyMedType << " < " << slowestMedType);
                if (   (bestCopyIter == m_cpyMedPos_v.end())
                    || (bestCopyMedType < slowestMedType)
                    || (  (bestCopyMedType == slowestMedType)
                        && bestCopySplitCnt > Int32_t(copyIter->size())
                       )
                   ) {
                    bestCopyIter     = copyIter;
                    bestCopyMedType  = slowestMedType;
                    bestCopySplitCnt = copyIter->size();
                }
            }
nextCopy: ;
        } // for (; copyIter != m_cpyMedPos_v.end(); ++copyIter)
        log_DBG_m(dbg_DETAIL, "Good copies per pool " << goodCopiesPerPool);


        if (goodCopiesPerPool < 2) { // no redundant copy yet
            if (bestCopyIter == m_cpyMedPos_v.end()) {
                if(m_best_effort){
                    m_numOfPurgedCopies++;
                    ostringstream sstr;
                    sstr << "Missing generation copy skipped on best effort base for :"
                         << "\n  fileID      " << m_curFileID
                         << "\n  migrationID " << m_migID
                         << "\n  stream size " << m_streamSize;
                    log_WriteEvent(sstr.str(), "COPY CONTENT");
                }
                else {
                    m_numOfUnavaillCopies++;
                    // warning no data source available
                    log_WRN_m("No useful copy for : "
                            << "\n  fileID      " << m_curFileID
                            << "\n  migrationID " << m_migID
                            << "\n  stream size " << m_streamSize);
                }
                if (m_currentVolume > 0){
                    while (m_unavailFilesPerVol.size() < m_currentVolume) {
                        m_unavailFilesPerVol.push_back(0);
                    }
                    m_unavailFilesPerVol[m_currentVolume-1]++;

                    log_DBG_m(dbg_DETAIL, "Increasing counter for unavailable files "
                               " for volume " << m_currentVolume
                            << " to " << m_unavailFilesPerVol[m_currentVolume-1]);
                }

                return true;
            }
            m_numOfRedundCopies++;
            log_DBG_m(dbg_DETAIL, "Got useful copy for : "
                           "\n  fileID      " << m_curFileID
                        << "\n  migrationID " << m_migID
                        << "\n  stream size " << m_streamSize
                        << "\n on media: ");

            ivd_MediaPos_v_i mpIter = bestCopyIter->begin();
            ivd_MediaPos_v_i  mpEnd = bestCopyIter->end();
            for (; mpIter != mpEnd; ++mpIter) {
                ivd_MediaPos_t &medPos = *mpIter;
                log_DBG_m(dbg_DETAIL, "  medKey     " << medPos.mediaKey
                                    << "\n  medVolNum  " << medPos.medVolNum
                                    << "\n  blockOffset" << medPos.blockOffset
                                    << "\n  splitOffset" << medPos.splitOffset
                                    << "\n  splitSize  " << medPos.splitSize);
                UInt64_t key = KeyGen(medPos.mediaKey, medPos.medVolNum);

                fsc_RedunCopyDataVolStorage *dataStor
                        = (fsc_RedunCopyDataVolStorage*)(m_cache.Get(key));
                if (dataStor == NULL) {
                    dataStor = new fsc_RedunCopyDataVolStorageWriter(
                                        key,
                                        medPos.mediaKey,
                                        medPos.medVolNum,
                                        m_results);
                    m_cache.Insert(dataStor);
                }
                ivd_FileLocationData_t  locData;
                locData.blockOffset = medPos.blockOffset;
                locData.fileID      = m_curFileID;
                locData.migrationID = m_migID;
                locData.splitSize   = medPos.splitSize;
                dataStor->WriteToBuffer(locData);
            }
        }
        else {
            m_numOfSkippedCopies++;
        }
    }
    return true;
}

Here is the call graph for this function:

bool fsc_CollectorRedunCopy::PostRecCopy ( fsc_Copy_t a_copy  )  [virtual]

Called after each copy processed.

Reimplemented from fsc_RawCollector.

Definition at line 121 of file fsc_CollectorRedunCopy.cpp.

References log_FUNC_m, and m_copyIter.

                                                           {
    log_FUNC_m(PostRecCopy);
    ++m_copyIter; // next copy
    return true;
}

bool fsc_CollectorRedunCopy::ProcRecGeneration ( fsc_Generation_t a_gen,
UInt32_t  a_status 
) [virtual]
bool fsc_CollectorRedunCopy::ProcRecMedPosMgr ( fsc_MediaPosition_t a_medPos  )  [virtual]

Called when media position is known.

Reimplemented from fsc_RawCollector.

Definition at line 136 of file fsc_CollectorRedunCopy.cpp.

References fsc_MediaPosition_t::blockOffset, ivd_MediaPos_t::blockOffset, ivd_MediaPos_t::dataType, dbg_DETAIL, log_DBG_m, log_FUNC_m, m_copyIter, m_dataOffset, m_dataSize, m_doRedunCopy, m_origMedKey, m_origMedVol, m_streamSize, fsc_MediaPosition_t::mediaKey, ivd_MediaPos_t::mediaKey, fsc_MediaPosition_t::medVolNum, ivd_MediaPos_t::medVolNum, ivd_MediaPos_t::splitOffset, and ivd_MediaPos_t::splitSize.

                                                                           {
    log_FUNC_m(ProcRecMedPosMgr);

    ivd_MediaPos_t medPos;

    medPos.dataType   = 0; // doesn't matter
    medPos.medVolNum  = a_medPos.medVolNum;
    medPos.mediaKey   = a_medPos.mediaKey;
    medPos.blockOffset= a_medPos.blockOffset;
    medPos.splitOffset= m_dataOffset;
    medPos.splitSize  = m_dataSize;

    m_copyIter->push_back(medPos);

    if (!m_doRedunCopy) {

        for (UInt32_t i(0); i < m_origMedVol.size(); i++){
            log_DBG_m(dbg_DETAIL, "m_origMedVol[i] " << m_origMedVol[i] <<
                                  " medPos.medVolNum " << medPos.medVolNum);
            if ( (m_origMedVol[i] == medPos.medVolNum) &&
                 (m_origMedKey == medPos.mediaKey) ) {
                m_doRedunCopy = true;
            }
        }

        log_DBG_m(dbg_DETAIL, " a_medPos.mediaKey == m_origMedKey "
                        << a_medPos.mediaKey << " == " << m_origMedKey
                        << " Do redundant copy " << boolalpha << m_doRedunCopy);
    }
    m_dataSize    = m_streamSize;
    m_dataOffset  = 0;
    return true;
}

bool fsc_CollectorRedunCopy::ProcRecSplit ( fsc_Split_t a_split  )  [virtual]

Called when generation or copy has splits.

Reimplemented from fsc_RawCollector.

Definition at line 128 of file fsc_CollectorRedunCopy.cpp.

References fsc_Split_t::dataOffset, fsc_Split_t::dataSize, log_FUNC_m, m_dataOffset, and m_dataSize.

                                                              {
    log_FUNC_m(ProcRecSplit);
    m_dataOffset = a_split.dataOffset;
    m_dataSize   = a_split.dataSize;
    return true;
}


Member Data Documentation

Reimplemented from fsc_RawCollector.

Definition at line 79 of file fsc_CollectorRedunCopy.h.

do redundant copy collection on best effort base

Definition at line 129 of file fsc_CollectorRedunCopy.h.

Referenced by PostCopiesProc(), and ~fsc_CollectorRedunCopy().

MedVolume file storage.

Definition at line 98 of file fsc_CollectorRedunCopy.h.

Referenced by PostCopiesProc().

Definition at line 125 of file fsc_CollectorRedunCopy.h.

Referenced by PostRecCopy(), ProcRecGeneration(), and ProcRecMedPosMgr().

each generation clear and fill its media position

Definition at line 124 of file fsc_CollectorRedunCopy.h.

Referenced by PostCopiesProc(), and ProcRecGeneration().

Definition at line 92 of file fsc_CollectorRedunCopy.h.

Referenced by PostCopiesProc().

generation and MedPos set to 0, split set its value

Definition at line 118 of file fsc_CollectorRedunCopy.h.

Referenced by ProcRecGeneration(), ProcRecMedPosMgr(), and ProcRecSplit().

generation and MedPos set to m_streamSize, split set its value

Definition at line 121 of file fsc_CollectorRedunCopy.h.

Referenced by ProcRecGeneration(), ProcRecMedPosMgr(), and ProcRecSplit().

each generation set this member

Definition at line 115 of file fsc_CollectorRedunCopy.h.

Referenced by ProcRecGeneration().

Set if copy exists on original medium, reset for each gen.

Definition at line 132 of file fsc_CollectorRedunCopy.h.

Referenced by PostCopiesProc(), ProcRecGeneration(), and ProcRecMedPosMgr().

Definition at line 126 of file fsc_CollectorRedunCopy.h.

Definition at line 89 of file fsc_CollectorRedunCopy.h.

Referenced by PostCopiesProc().

Definition at line 109 of file fsc_CollectorRedunCopy.h.

Referenced by PostCopiesProc(), and ProcRecGeneration().

Definition at line 105 of file fsc_CollectorRedunCopy.h.

Referenced by PostCopiesProc(), and ~fsc_CollectorRedunCopy().

Definition at line 101 of file fsc_CollectorRedunCopy.h.

Referenced by PostCopiesProc(), and ~fsc_CollectorRedunCopy().

Definition at line 107 of file fsc_CollectorRedunCopy.h.

Referenced by PostCopiesProc(), and ~fsc_CollectorRedunCopy().

Definition at line 103 of file fsc_CollectorRedunCopy.h.

Referenced by PostCopiesProc(), and ~fsc_CollectorRedunCopy().

Definition at line 91 of file fsc_CollectorRedunCopy.h.

Referenced by ProcRecMedPosMgr().

Definition at line 94 of file fsc_CollectorRedunCopy.h.

Referenced by PostCopiesProc().

Definition at line 95 of file fsc_CollectorRedunCopy.h.

Referenced by fsc_CollectorRedunCopy().

Definition at line 86 of file fsc_CollectorRedunCopy.h.

Referenced by PostCopiesProc().

stream size stored in generation

Definition at line 112 of file fsc_CollectorRedunCopy.h.

Referenced by PostCopiesProc(), ProcRecGeneration(), and ProcRecMedPosMgr().

Definition at line 93 of file fsc_CollectorRedunCopy.h.

Referenced by PostCopiesProc().


The documentation for this class was generated from the following files: