Public Member Functions | Public Attributes | Private Member Functions | Private Attributes

hsm_FHmigc Class Reference
[Classes for managing object list]

hsm_FHmigc class manages files which are "Dirty". More...

#include <hsm_FHmigc.h>

Inheritance diagram for hsm_FHmigc:
Inheritance graph
[legend]
Collaboration diagram for hsm_FHmigc:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 hsm_FHmigc (bool &a_running, i_PartitionManager_ptr a_iPM, ivd_Time32_t a_minWaitTime, UInt32_t a_minNumFiles, ivd_FileSize_t a_minMigSize, ivd_Time32_t a_maxWaitTime, UInt32_t a_maxNumFiles, ivd_FileSize_t a_maxMigSize, hsm_MigByAPI &a_migJobs)
virtual ~hsm_FHmigc ()
virtual hsm_FH_p_li Append (hsm_FileHeader *a_fh_p)
 list methods
void Remove (hsm_FileHeader *a_fh_p)
ivd_FileSize_t GetFilesListSize ()
void Reconfigure (ivd_Time32_t a_CfgMinWaitTime, UInt32_t a_CfgMinNumFiles, ivd_FileSize_t a_CfgMinMigSize, ivd_Time32_t a_CfgMaxWaitTime, UInt32_t a_CfgMaxNumFiles, ivd_FileSize_t a_CfgMaxMigSize)
ivd_Time32_t GetWakeUpTime ()
UInt32_t TrigMigration (void)
virtual void Run (void *arg)
void Shutdown ()
void WakeUp (void)
void Migrate (UInt32_t jobSize)
void SendToPM (UInt32_t sizeTL, UInt32_t filesNoFID, UInt32_t preJobIdx)
bool GetFileIDs (UInt32_t filesNoFID, i_FileIDs_t_var &tFIDs)
void DumpStatus (ostringstream &sstr)
void DumpList (ostringstream &sstr)

Public Attributes

 log_CLASSID_m

Private Member Functions

void MigrateParents (hsm_FileHeader *a_fh_p, UInt32_t &a_filesNoFID, UInt32_t &a_fileCount, ivd_FileSize_t &a_migDataSize, const UInt32_t a_preJobIdx)
bool ShouldMigrate ()

Private Attributes

i_PartitionManager_var m_iPM
bool & m_running
 outside variable that set when thread is down
bool m_goDown
cmn_Mutex m_thread_x
cmn_Condition m_thread_c
ivd_Time32_t m_minWaitTime
 If nothing change in minWaitTime perion and see m_minNumMigFile, m_minMigSize.
UInt32_t m_minNumFiles
 at least this number of files to be mig at minWaitTime
ivd_FileSize_t m_minMigSize
 Amount of data that has to be achive to migrate in minWaitTime.
ivd_Time32_t m_maxWaitTime
 If maxWaitTime passed from last migration then mig now.
UInt32_t m_maxNumFiles
 If m_maxNumFiles lay on list then migrate immediately.
ivd_FileSize_t m_maxMigSize
 If m_maxMigSize data is add to mig list then migrate it.
hsm_MigByAPIm_migJobs
ivd_Time32_t m_wakeUpTime
 thread wake up time
ivd_FileSize_t m_migDataSize
ivd_Time32_t m_firstAppendTime
ivd_Time32_t m_lastAppendTime
bool m_forcedMigration
UInt64_t m_majColId
UInt64_t m_minColId
UInt32_t m_flags

Detailed Description

hsm_FHmigc class manages files which are "Dirty".

Definition at line 116 of file hsm_FHmigc.h.


Constructor & Destructor Documentation

hsm_FHmigc::hsm_FHmigc ( bool &  a_running,
i_PartitionManager_ptr  a_iPM,
ivd_Time32_t  a_minWaitTime,
UInt32_t  a_minNumFiles,
ivd_FileSize_t  a_minMigSize,
ivd_Time32_t  a_maxWaitTime,
UInt32_t  a_maxNumFiles,
ivd_FileSize_t  a_maxMigSize,
hsm_MigByAPI a_migJobs 
)

Definition at line 63 of file hsm_FHmigc.cpp.

References dbg_LOW, log_DBG_m, log_FUNC_m, m_iPM, m_maxMigSize, m_maxNumFiles, m_maxWaitTime, m_minMigSize, m_minNumFiles, and m_minWaitTime.

        :
        m_running(a_running),
        m_goDown(false),
        m_thread_c(&m_thread_x),
        m_minWaitTime(a_minWaitTime),
        m_minNumFiles(a_minNumFiles),
        m_minMigSize(a_minMigSize),
        m_maxWaitTime(a_maxWaitTime),
        m_maxNumFiles(a_maxNumFiles),
        m_maxMigSize(a_maxMigSize),
        m_migJobs(a_migJobs),
        m_wakeUpTime(0),
        m_migDataSize(0),
        m_firstAppendTime(0),
        m_lastAppendTime(0),
        m_forcedMigration(false),
        m_majColId(0),
        m_minColId(0),
        m_flags(0)

{
    log_FUNC_m(hsm_FHmigc);

    log_DBG_m(dbg_LOW, "Create migration candidate list." << endl <<
        "-----------------------------------------------------------------------" << endl <<
        "MinWaitTime "   << setw(8)  << m_minWaitTime <<
        "\tMinNumFiles " << setw(8)  << m_minNumFiles <<
        "\tMinMigSize "  << setw(10) << m_minMigSize  << endl <<
        "MaxWaitTime "   << setw(8)  << m_maxWaitTime <<
        "\tMaxNumFiles " << setw(8)  << m_maxNumFiles <<
        "\tMaxMigSize "  << setw(10) << m_maxMigSize  << endl <<
        "-----------------------------------------------------------------------");
    m_iPM = i_PartitionManager::_duplicate(a_iPM);

}

virtual hsm_FHmigc::~hsm_FHmigc (  )  [inline, virtual]

Definition at line 131 of file hsm_FHmigc.h.

{};


Member Function Documentation

hsm_FH_p_li hsm_FHmigc::Append ( hsm_FileHeader a_fh_p  )  [virtual]

list methods

Definition at line 111 of file hsm_FHmigc.cpp.

References hsm_FileHeader::GetFileSizeForMig(), hsm_FileHeader::IncrRef(), log_FUNC_m, hsm_FHlist::m_FHlist_x, m_firstAppendTime, m_lastAppendTime, m_migDataSize, NULL, ShouldMigrate(), and WakeUp().

Referenced by hsm_FileHeader::ToMigCand().

                                                     {
    log_FUNC_m(Append);
    a_fh_p->IncrRef();
    hsm_FH_p_li iter;
    {
        cmn_MutexLock l(m_FHlist_x);

        // check if file is deletted is implement inside GetFileSizeForMig() method
        m_migDataSize += a_fh_p->GetFileSizeForMig();

        m_lastAppendTime = time(NULL);
        if (m_firstAppendTime == 0) {
            m_firstAppendTime = m_lastAppendTime;
        }
        iter = insert(end(), a_fh_p);
    }

    if (ShouldMigrate()) {
        WakeUp();
    }

    return iter;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void hsm_FHmigc::DumpList ( ostringstream &  sstr  ) 

Definition at line 861 of file hsm_FHmigc.cpp.

References hsm_FileHeader::DumpListEl(), hsm_FHlist::m_FHlist_x, m_goDown, NULL, and size.

Referenced by i_HSM_i::ShowStatus().

                                             {
    cmn_MutexLock l(m_FHlist_x);

    sstr << endl << "MIG Candidate List Contents" << endl;
    sstr << "ChgFlg     FileID               Inode   Name... Old name" << endl;
    sstr << "------------------------------------------------------------" << endl;
    hsm_FH_p_li iter = begin();
    for (;iter != end(); ++iter) {
        if (m_goDown) {
            sstr << "Shut down. Abort Dirty List." << endl;
            break;
        }
        hsm_FileHeader &fh = (**iter);
        if (&fh == NULL) {
            sstr << "NULL pointer found." << endl;
            break;
        }
        fh.DumpListEl(sstr);
        if (sstr.str().size() > FSC_MAX_CORBA_BUFFERSIZE) {
            sstr << "Stop dumping. Too much elements " << size()
                 << ". Buff size " << sstr.str().size() << endl;
            break;
        }
    }
    sstr << "Total files " << size() << endl;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void hsm_FHmigc::DumpStatus ( ostringstream &  sstr  ) 

Definition at line 831 of file hsm_FHmigc.cpp.

References cmn_Num2PowOfK(), GetFilesListSize(), GetWakeUpTime(), m_firstAppendTime, m_lastAppendTime, m_maxMigSize, m_maxNumFiles, m_maxWaitTime, m_minMigSize, m_minNumFiles, m_minWaitTime, size, cmn_Time::Time2hms(), and cmn_Time::Time2YMDhms().

Referenced by i_HSM_i::ShowStatus().

                                               {
    cmn_Time upMtime(GetWakeUpTime() ,0);
    cmn_Time firstAppendTime(m_firstAppendTime);
    cmn_Time lastAppendTime(m_lastAppendTime);
    cmn_Time minWaitTime(m_minWaitTime, 0, true);
    cmn_Time maxWaitTime(m_maxWaitTime, 0, true);

    sstr << "Number of files waiting for migration:  " << size() << endl;
    sstr << "Size of files waiting for migration:    " << cmn_Num2PowOfK(GetFilesListSize()) << endl;
    sstr << "Cfg: Max mig size                       " << m_maxMigSize << endl;
    sstr << "Cfg: Max number of files                " << m_maxNumFiles << endl;
    sstr << "Cfg: Min mig size                       " << m_minMigSize << endl;
    sstr << "Cfg: Min number of files                " << m_minNumFiles << endl;

    sstr << "Next Migration will wake up at:         " << upMtime.Time2YMDhms() << endl;
    sstr << "First append time:                      " << firstAppendTime.Time2YMDhms() << endl;
    sstr << "Last  append time:                      " << lastAppendTime.Time2YMDhms() << endl;
    sstr << "Cfg: min wait time:                     " << minWaitTime.Time2hms() << endl;
    sstr << "Cfg: max wait time:                     " << maxWaitTime.Time2hms() << endl;

               //   && m_wakeUpTime    >= m_lastAppendTime + m_minWaitTime
               //   && numFile       >= m_minNumFiles
               //   && m_migDataSize >= m_minMigSize )
               //|| m_wakeUpTime    >= m_firstAppendTime + m_maxWaitTime
               //|| numFile       >= m_maxNumFiles
               //|| m_migDataSize >= m_maxMigSize
               //|| m_forcedMigration) {
}

Here is the call graph for this function:

Here is the caller graph for this function:

bool hsm_FHmigc::GetFileIDs ( UInt32_t  filesNoFID,
i_FileIDs_t_var &  tFIDs 
)

Definition at line 712 of file hsm_FHmigc.cpp.

References dbg_DETAIL, fio_Transaction::EndTransaction(), evt_ERROR, fio_RelFile::FirstNewIDX(), g_File2hdbID_p, g_hsmDB_p, ivd_BaseException::GetDescription(), ivd_BaseException::GetFriendly(), fio_DataBase::GetTransObj(), ipc_EXEC_m, log_DBG_m, log_ERR_m, log_FUNC_m, log_WriteEvent(), m_iPM, fio_DataBase::ReleaseTransObj(), fio_Transaction::StartTransaction(), and fio_RelFile::WriteRec().

Referenced by SendToPM().

                                                                       {
    log_FUNC_m(GetFileIDs);

    if (filesNoFID == 0) {
        return true;
    }

    i_FSC_var tFSC;

    log_DBG_m(dbg_DETAIL, "  MigList ask PM to get FSC");

    try {
        ipc_EXEC_m(
            tFSC = m_iPM->GetFSC();
        );
    }
    catch (ivd_Exception &ie) {
        log_ERR_m("Caught Corba or HSM exception: " + ie.GetDescription());
        return false;
    }
    catch (...){
        log_ERR_m("Caught other exception.");
        return false;
    }

    if( CORBA::is_nil(tFSC) ) {
        log_ERR_m("i_FSC is_nil!");
        return false;
    }
//        log_DBG_m(dbg_DETAIL, "  MigList called FSC to get " << filesNumFID << " FileIDs");
    try {
        ipc_EXEC_m(
            (tFIDs) = tFSC->GetFileIDs(filesNoFID);
        );
    }
    catch (ivd_Exception &ie) {
        log_ERR_m("FSC->GetFileIDs() failed: " + ie.GetFriendly());
        ostringstream sstr;
        sstr << "Failed to allocate File IDs from FSC ("
                << ie.GetFriendly() << ").";
        log_WriteEvent(evt_ERROR, sstr.str());
        return false;
    }
    catch (...){
        log_ERR_m("FSC->GetFileIDs() failed: unknown exception");
        log_WriteEvent(evt_ERROR, "Failed to allocate File IDs from FSC.");
        return false;
    }

    log_DBG_m(dbg_DETAIL, "  FSC return " << (tFIDs)->length() <<
                            " elements");

    if (filesNoFID != (tFIDs)->length()){
        return false;
//            UInt32_t numids = tFIDs->length();
//            throw ivd_InternalError(ie_EMPTYLIST ,"Number of returned FileID's " + cmn_Num2Str(numids) + " are not as requested " + cmn_Num2Str(filesNumFID));
    }
    //     Save max FileID;
    ivd_RecordIDX_t maxFileID = 0;
    i_FileIDs_t &fidseq = (tFIDs);

    int i = fidseq.length();
    while (i > 0) {
        --i;
        if (maxFileID < fidseq[i]) {
            maxFileID = fidseq[i];
        }
    }
    //
    ostringstream ostr;
    ostr << "Last fileID: " << maxFileID;
    log_WriteEvent(ostr.str(), "FSC");

    ivd_RecordIDX_t hdbMaxFileID = g_File2hdbID_p->FirstNewIDX();
    if (hdbMaxFileID <= maxFileID) {
        log_DBG_m(dbg_DETAIL, "Extend file2hdbID file up to idx " << maxFileID
                            << " prevoius max fileID = " << hdbMaxFileID);
        fio_Transaction *trans = g_hsmDB_p->GetTransObj();
        trans->StartTransaction();
        hdb_file2hdbID_t file2hdbID; // an empty record
        g_File2hdbID_p->WriteRec(maxFileID, &file2hdbID, 1, trans);
        trans->EndTransaction();
        g_hsmDB_p->ReleaseTransObj(&trans);
        log_DBG_m(dbg_DETAIL, "Max fileID after extenson = " << g_File2hdbID_p->FirstNewIDX());
    }

    return true;
}

Here is the call graph for this function:

Here is the caller graph for this function:

ivd_FileSize_t hsm_FHmigc::GetFilesListSize (  )  [inline]

Definition at line 193 of file hsm_FHmigc.h.

References m_migDataSize.

Referenced by DumpStatus(), and i_HSM_i::GetInfo().

{return m_migDataSize;};

Here is the caller graph for this function:

ivd_Time32_t hsm_FHmigc::GetWakeUpTime (  )  [inline]

Definition at line 201 of file hsm_FHmigc.h.

References m_wakeUpTime.

Referenced by DumpStatus(), and i_HSM_i::GetInfo().

{return m_wakeUpTime;};

Here is the caller graph for this function:

void hsm_FHmigc::Migrate ( UInt32_t  jobSize  ) 

Definition at line 293 of file hsm_FHmigc.cpp.

References ClientConf_t::CfgMigrateWithCollocation, dbg_DETAIL, dbg_LOW, dbg_NORM, g_clientConf_p, g_hsm_fhLock, g_hsm_preJobList_p, g_mig, hsm_MigByAPI::GetAPIMode(), ClientConf_t::GetCollocationId(), hsm_FileHeader::GetFileID(), hsm_FileHeader::GetFileSizeForMig(), hsm_FileHeader::GetFileType(), hsm_FileHeader::GetFOwner(), hsm_FileHeader::GetFullPath(), hsm_FileHeader::GetInode(), hsm_JobListMgr::GetJobIdx(), hsm_FileHeader::GetOldFullPath(), hjs_PREPARATION, ie_NULLPTR, hsm_FileHeader::IsDeleted(), IVD_PRINT_ID_FS, cmn_Mutex::Lock(), log_DBG_m, log_FUNC_m, log_NOTE_m, hsm_FHlist::m_FHlist_x, m_goDown, m_majColId, m_maxMigSize, m_maxNumFiles, m_migDataSize, m_minColId, hsm_FileHeader::MigCanToPreJob(), MigrateParents(), NULL, hsm_JobListMgr::ReleaseJobIdx(), hsm_FileHeader::RemoveFromHSM(), SendToPM(), hsm_JobListMgr::SetJobStatus(), ShouldMigrate(), cmn_Mutex::Unlock(), and hsm_MigByAPI::WasFileRequested().

Referenced by Run().

                                         {
    log_FUNC_m(Migrate);

    do {
        UInt32_t preJobIdx  = g_hsm_preJobList_p->GetJobIdx(jobSize);
        log_DBG_m(dbg_LOW, "preJobIdx: " << preJobIdx << ", jobSize: " << jobSize);
        g_hsm_preJobList_p->SetJobStatus(preJobIdx, hjs_PREPARATION);
        UInt32_t count      = 0;
        UInt32_t filesNoFID = 0;

        ivd_FileSize_t migDataSize = 0;

        m_FHlist_x.Lock();
        hsm_FH_p_li iter = begin();
        bool firstFile(true);
        UInt32_t numberOfSkippedFiles(0);
        while (  iter  != end()
            && count <= jobSize) {
            if (m_goDown) {
                m_FHlist_x.Unlock();
                g_hsm_preJobList_p->ReleaseJobIdx(preJobIdx);
                return;
            }

            hsm_FileHeader &fh = (**iter);
            if (&fh == NULL) {
                log_FUNC_m(MngDirty);
                throw ivd_InternalError(ie_NULLPTR, "Mig candidate list contain NULL pointer.", true);
            }

            //  try to lock FH by inode
            ivd_GenInode_t inode = fh.GetInode();
            if (!g_hsm_fhLock.CanLockByID(inode)) {
                iter++;
                continue;  // can't use this FH already locked, so skip it
            }
            // now FH is locked, so list can be unlock
            m_FHlist_x.Unlock();

            if (   g_mig.GetAPIMode()
                && (fh.GetFileType() == ift_FILE)
                && (!fh.IsDeleted())
                && (!g_mig.WasFileRequested(inode))) {

                log_NOTE_m("File " << IVD_PRINT_ID_FS(inode) << " migration was not requested by API.");
            }

            bool addFileToJob(true);
            bool removedFromDb(false);
            if (g_clientConf_p->CfgMigrateWithCollocation) {
                addFileToJob = false;
                cmn_Path fullPath(fh.GetFullPath());
                if (fullPath.empty()) {
                    log_DBG_m(dbg_DETAIL, "Using old path to determine collocation ID.");
                    fullPath = fh.GetOldFullPath();
                }

                UInt64_t colId(0);
                try {
                    colId = g_clientConf_p->GetCollocationId(fullPath);
                }
                catch (...) {
                    log_DBG_m(dbg_LOW, "Cannot get collocation ID for " << fullPath);
                }

                log_DBG_m(dbg_DETAIL, "File colId: " << colId << " " << fullPath);

                if ((colId != 0) || fh.IsDeleted()) {
                    if (firstFile) {
                        log_DBG_m(dbg_DETAIL, "Job colId: " << colId);
                        m_majColId = colId;
                        m_minColId = colId;
                        firstFile = false;
                    }

                    addFileToJob = (   (m_majColId == colId)
                                    || ((colId == 0) && fh.IsDeleted()));
                }
                else {
                    fh.RemoveFromHSM();
                    log_DBG_m(dbg_DETAIL, "Removed from HSMDB: " << fullPath);
                    removedFromDb = true;
                }
            } // if (g_clientConf_p->CfgMigrateWithCollocation) {

            if (addFileToJob) {
                // Check if all parent directories are migrated
                MigrateParents( fh.GetFOwner(),
                                filesNoFID,
                                count,
                                migDataSize,
                                preJobIdx);

                if (fh.GetFileID() == 0) {  // candidate for fileID
                    filesNoFID++;
                }

                fh.MigCanToPreJob(preJobIdx);
                count++;
                migDataSize += fh.GetFileSizeForMig();
            }
            else if (!removedFromDb) {
                numberOfSkippedFiles++;
            }

            g_hsm_fhLock.UnLockByID(inode);

            if (   migDataSize >= m_maxMigSize
                || count       >= m_maxNumFiles) {
                log_DBG_m(dbg_NORM, "Job is large enough.  DataSize "  <<  m_migDataSize
                                << " number of files " << count);
                goto sendit;
            }

            m_FHlist_x.Lock();
            iter = begin();
            for (UInt32_t i(0); i < numberOfSkippedFiles; i++) {
                iter++;
            }
        }

        m_FHlist_x.Unlock();

    sendit:
        if (count > 0) {
            SendToPM(count, filesNoFID, preJobIdx);
        }
        else {
            g_hsm_preJobList_p->ReleaseJobIdx(preJobIdx);
            log_DBG_m(dbg_DETAIL, "No more files to do preJob.");
        }

    } while (ShouldMigrate());

}

Here is the call graph for this function:

Here is the caller graph for this function:

void hsm_FHmigc::MigrateParents ( hsm_FileHeader a_fh_p,
UInt32_t a_filesNoFID,
UInt32_t a_fileCount,
ivd_FileSize_t a_migDataSize,
const UInt32_t  a_preJobIdx 
) [private]

Definition at line 431 of file hsm_FHmigc.cpp.

References dbg_DETAIL, dbg_NORM, hsm_FileHeader::GetFileID(), hsm_FileHeader::GetFileSizeForMig(), hsm_FileHeader::GetFOwner(), hsm_FileHeader::GetFullPath(), hsm_FileHeader::GetList(), log_DBG_m, log_FUNC_m, hsm_FileHeader::MigCanToPreJob(), NULL, and hsm_FileHeader::ToMigCand().

Referenced by Migrate().

                                                            {
    log_FUNC_m(MigrateParents);

    if (a_fh_p == NULL) {
        log_DBG_m(dbg_DETAIL, "Null");
        return;
    }

    if (   (a_fh_p->GetFileID() == 0)
        && !(a_fh_p->GetFullPath().empty())) {

        MigrateParents( a_fh_p->GetFOwner(),
                        a_filesNoFID,
                        a_fileCount,
                        a_migDataSize,
                        a_preJobIdx);

        if (a_fh_p->GetList() != ePreJobList) {
            log_DBG_m(dbg_NORM, "Adding dirty directory " << a_fh_p->GetFullPath() << " to job.");

            a_filesNoFID++;

            a_fh_p->ToMigCand();
            a_fh_p->MigCanToPreJob(a_preJobIdx);
            a_fileCount++;
            a_migDataSize += a_fh_p->GetFileSizeForMig();
        }
        else {
            log_DBG_m(dbg_DETAIL, "Dirty directory " << a_fh_p->GetFullPath() << " already on job list.");
        }
    }

}

Here is the call graph for this function:

Here is the caller graph for this function:

void hsm_FHmigc::Reconfigure ( ivd_Time32_t  a_CfgMinWaitTime,
UInt32_t  a_CfgMinNumFiles,
ivd_FileSize_t  a_CfgMinMigSize,
ivd_Time32_t  a_CfgMaxWaitTime,
UInt32_t  a_CfgMaxNumFiles,
ivd_FileSize_t  a_CfgMaxMigSize 
)

Definition at line 804 of file hsm_FHmigc.cpp.

References dbg_LOW, log_DBG_m, log_FUNC_m, m_maxMigSize, m_maxNumFiles, m_maxWaitTime, m_minMigSize, m_minNumFiles, and m_minWaitTime.

Referenced by i_HSM_i::Reconfigure().

                                                                  {
    log_FUNC_m(Reconfigure);

    m_minWaitTime   = a_CfgMinWaitTime;
    m_minNumFiles   = a_CfgMinNumFiles;
    m_minMigSize    = a_CfgMinMigSize;
    m_maxWaitTime   = a_CfgMaxWaitTime;
    m_maxNumFiles   = a_CfgMaxNumFiles;
    m_maxMigSize    = a_CfgMaxMigSize;

    log_DBG_m(dbg_LOW, "Reconfigure migration candidate list." << endl <<
        "-----------------------------------------------------------------------" << endl <<
        "MinWaitTime "   << setw(8)  << m_minWaitTime <<
        "\tMinNumFiles " << setw(8)  << m_minNumFiles <<
        "\tMinMigSize "  << setw(10) << m_minMigSize  << endl <<
        "MaxWaitTime "   << setw(8)  << m_maxWaitTime <<
        "\tMaxNumFiles " << setw(8)  << m_maxNumFiles <<
        "\tMaxMigSize "  << setw(10) << m_maxMigSize  << endl <<
        "-----------------------------------------------------------------------");
}

Here is the caller graph for this function:

void hsm_FHmigc::Remove ( hsm_FileHeader a_fh_p  ) 

Definition at line 137 of file hsm_FHmigc.cpp.

References hsm_FileHeader::DecrRef(), hsm_FileHeader::GetFileSizeForMig(), hsm_ListPos::GetMigCanPos(), log_FUNC_m, hsm_FHlist::m_FHlist_x, and m_migDataSize.

Referenced by hsm_ListPos::RemoveFromList().

                                              { // hsm_FH_p_li &a_pos
    log_FUNC_m(Remove);
    cmn_MutexLock l(m_FHlist_x);

    m_migDataSize -= a_fh_p->GetFileSizeForMig();
    a_fh_p->DecrRef();
    erase(a_fh_p->GetMigCanPos());
}

Here is the call graph for this function:

Here is the caller graph for this function:

void hsm_FHmigc::Run ( void *  arg  )  [virtual]

Wait MinWaitTime or wait to broadcast

Reimplemented from cmn_Thread.

Definition at line 166 of file hsm_FHmigc.cpp.

References cmn_Condition::Broadcast(), cmn_Global::dbg, dbg_DETAIL, dbg_NORM, errno, g_cmn, g_jobList_c, g_jobList_x, g_mig, hsm_MigByAPI::GetAPIMode(), log_Debugger::GetLevel(), cmn_Mutex::Lock(), log_DBG_m, log_FUNC_m, hsm_FHlist::m_FHlist_x, m_firstAppendTime, m_forcedMigration, m_goDown, m_lastAppendTime, m_maxWaitTime, m_migDataSize, m_minWaitTime, m_running, m_thread_c, m_thread_x, m_wakeUpTime, Migrate(), NULL, ShouldMigrate(), size, cmn_Time::Time2hms(), cmn_Condition::TimedWait(), and cmn_Mutex::Unlock().

                             {
    log_FUNC_m(Run);
    m_running = true;

    log_DBG_m(dbg_DETAIL, "Migration cand. thread is starting up");

    while (!m_goDown) {

        if (g_mig.GetAPIMode()){
            //recall only mode
            //let migration candidate list migrate once a day
            log_DBG_m(dbg_NORM, "minWaitTime set to 24h");
            m_minWaitTime = cfg_DAY;
        }

        m_wakeUpTime = time(NULL) + m_minWaitTime;

        {
            cmn_Time cmntimewakeup(m_wakeUpTime, 0);
            log_DBG_m(dbg_DETAIL, "MigCndLThread will wake up at latest "
                                << cmntimewakeup.Time2hms());
        }

        try {
            m_thread_x.Lock();

            int retv = m_thread_c.TimedWait(m_wakeUpTime);
            m_thread_x.Unlock();

            if (m_goDown) {
                break;
            }

            UInt32_t numFile;
            {
                cmn_MutexLock listLock(m_FHlist_x);
                numFile = size();

                if (numFile == 0) {
                    log_DBG_m(dbg_DETAIL, "NO files to migrate");

                    //Broadcast to thread waiting for triggered migrations
                    log_DBG_m(dbg_DETAIL, "before Lock");
                    cmn_MutexLock l(g_jobList_x);
                    g_jobList_c.Broadcast();

                    continue;
                }

                if (g_cmn.dbg.GetLevel() == dbg_DETAIL) {
                    cmn_Time firstAppTime(m_firstAppendTime);

                    cmn_Time lastAppTime(m_lastAppendTime);

                    log_DBG_m(dbg_DETAIL,
                          "MigCndLThread properties of List: " << endl <<
                          "\tlast append time " << lastAppTime.Time2hms() <<
                          "\tminWaitTime "      << m_minWaitTime <<
                          " \tNumFiles "        << numFile << endl <<
                          "\tfirst append time " << firstAppTime.Time2hms() <<
                          "\tmaxWaitTime "      << m_maxWaitTime <<
                          " \tMigSize  "        << m_migDataSize <<
                          ((retv == 0) ? " TIMEOUT" : " BROADCAST"));
                    log_DBG_m(dbg_DETAIL, "APIMode() " << boolalpha << g_mig.GetAPIMode());
                }
            }

            if (   !g_mig.GetAPIMode()
                && (ShouldMigrate() || m_forcedMigration)) {

                m_forcedMigration = false;
                // reset list kumulative values
                m_firstAppendTime = 0;
                log_DBG_m(dbg_DETAIL, "MigCndLThread - Migration NEED to be started!");
                Migrate(numFile);
            } else if (g_mig.GetAPIMode()){
                m_forcedMigration = false;
                m_firstAppendTime = 0;
                log_DBG_m(dbg_DETAIL, "MigCndLThread - started by API");
                Migrate(numFile);

                //Broadcast to thread waiting for triggered migrations
                log_DBG_m(dbg_DETAIL, "before Lock");
                cmn_MutexLock l(g_jobList_x);
                g_jobList_c.Broadcast();
            }
            else {
                log_DBG_m(dbg_DETAIL, "MigCndLThread - Migration DO_NOT_NEED to be started yet!");
            }
        }
        catch (ivd_Error &ie) {
            log_DBG_m(dbg_DETAIL, "MigCndLThread return ERROR " << ie);
            // TODO manage lost old MigCnd List
        }
        catch (ivd_SysError &ise) {
            log_DBG_m(dbg_DETAIL, "MigCndLThread WILL GO_DOWN on ERROR " << ise);
            break;
        }
        catch (...) {
            log_DBG_m(dbg_DETAIL, "MigCndListMgrThrd WILL GO_DOWN after thrown ERROR. errno = " << errno);
            break;
        }
    } // while (!m_goDown)
    log_DBG_m(dbg_DETAIL, "Migration cand. thread IS_DOWN.");
    m_running = false;
}

Here is the call graph for this function:

void hsm_FHmigc::SendToPM ( UInt32_t  sizeTL,
UInt32_t  filesNoFID,
UInt32_t  preJobIdx 
)

Definition at line 501 of file hsm_FHmigc.cpp.

References hsm_MigByAPI::AddJob(), hsm_JobListMgr::At(), ivd_FS_File::Close(), cmn_Num2Str(), dbg_DETAIL, dbg_NORM, ivd_FS_File::e_Cache, fs_api::eDelFile, hsm_FileHeader::Event(), evt_ERROR, file, g_cmn, g_fs_api_p, g_hsm_fhLock, g_hsm_preJobList_p, g_hsmDB_p, g_jobList_x, g_mig, hsm_MigByAPI::GetAPIMode(), hsm_FileHeader::GetFileID(), GetFileIDs(), hsm_FileHeader::GetFName(), ivd_BaseException::GetFriendly(), hsm_FileHeader::GetInode(), ivd_FileSystemAPI::GetRootPath(), fio_DataBase::GetTransObj(), hjs_MIGFAILED, hjs_WAITFORDOMIG, hsm_MigByAPI::InsertJobInode(), ipc_EXEC_m, ivd_USleep, cmn_Mutex::Lock(), log_DBG_m, log_ERR_m, log_FUNC_m, log_WriteEvent(), m_flags, hsm_MigByAPI::m_flags, m_goDown, m_iPM, hsm_JobElemVecMgr::m_jobElemVecMgr_x, m_majColId, hsm_MigByAPI::m_majColId, m_minColId, hsm_MigByAPI::m_minColId, ivd_Product::m_nameShort, hsm_FileHeader::MakePath(), hsm_FileHeader::MigPreJobToPM(), NULL, ivd_FS_File::Open(), cmn_Global::prod, hsm_JobListMgr::ReleaseJobIdx(), fio_DataBase::ReleaseTransObj(), hsm_FileHeader::SetFileID(), hsm_JobListMgr::SetJobStatus(), hsm_JobListMgr::SetMigJobID(), and cmn_Mutex::Unlock().

Referenced by Migrate().

                                                {

    log_FUNC_m(SendToPM);
    int FileIDidx  = 0;
    UInt32_t    numfiles = 0;

    i_FileIDs_t_var tFIDs;
    if (!GetFileIDs(filesNoFID, tFIDs)) {
        log_ERR_m("Can't get fileIDs");
        return;
    }

    log_DBG_m(dbg_DETAIL, "  Sent To PM");

    log_DBG_m(dbg_DETAIL, "  To resolve " << sizeTL << " files for Migration");

    hsm_JobElemVecMgr &preJobVect = g_hsm_preJobList_p->At(a_preJobIdx);
    if (&preJobVect == NULL) {
        log_ERR_m("Can't get hsm_JobElemVecMgr");
        return;
    }

    list<hsm_FileHeader**> skipped;

    i_JobRequestList_t seqTFileList(sizeTL);
    seqTFileList.length(sizeTL);

    preJobVect.m_jobElemVecMgr_x.Lock();
    for (hsm_JobElem_v_i iter = preJobVect.begin(); iter != preJobVect.end(); iter++) {
        if (m_goDown) {
            preJobVect.m_jobElemVecMgr_x.Unlock();
            log_DBG_m (dbg_NORM, "Shut down. Abort SendToPM.");
            return;
        }

        hsm_FileHeader &fh = (**iter);
        if (&fh == NULL) {
            continue; // probably moved to dirty list
        }
        //  try to lock FH by inode
        ivd_GenInode_t inode = fh.GetInode();
        if (!g_hsm_fhLock.CanLockByID(inode)) {
            skipped.push_back(&*iter); // proceed latter
            continue;  // can't use this FH already locked, so skip it
        }
        preJobVect.m_jobElemVecMgr_x.Unlock();

        ivd_FileID_t fileID = fh.GetFileID();
        ivd_FS_File file(*g_fs_api_p, (g_fs_api_p->GetRootPath() + fh.MakePath()));

        if ( fileID == 0) {
            log_DBG_m (dbg_DETAIL, "SetFileID. tFIDs[" << FileIDidx << "]");
            try {
                //HPUX - The file needs to be opened (to be in the kernel cache)
                file.Open(ivd_FS_File::e_Cache);
                fh.SetFileID((tFIDs)[FileIDidx++]);
            }
            catch (ivd_SysError &ise) {
                // file is probably deleted in meantime
                log_DBG_m(dbg_NORM, "Try 1. File is probably deleted. Simulate delete event.");
                fio_Transaction &trans = *g_hsmDB_p->GetTransObj();
                cmn_Path oldName(fh.GetFName());
                fh.Event(fs_api::eDelFile, trans, &oldName);
                g_hsmDB_p->ReleaseTransObj(trans);

                g_hsm_fhLock.UnLockByID(inode);
                preJobVect.m_jobElemVecMgr_x.Lock();
                continue;
            }
        }
        fh.MigPreJobToPM(seqTFileList[numfiles++]);

        if ( g_mig.GetAPIMode() == true ){
            g_mig.InsertJobInode(fh.GetInode(), a_preJobIdx);
        }

        //HPUX Close file
        file.Close();
        g_hsm_fhLock.UnLockByID(inode);
        preJobVect.m_jobElemVecMgr_x.Lock();
    }
    preJobVect.m_jobElemVecMgr_x.Unlock();

    //  Proceed previous locked FH inode until are handled
    if (skipped.size() > 0) {
        log_DBG_m(dbg_DETAIL, "Some FH were locked. Try again.");
        ivd_USleep(10);
    // fix lock bug
        preJobVect.m_jobElemVecMgr_x.Lock();
        for (list<hsm_FileHeader**>::iterator iskip = skipped.begin();
             iskip  != skipped.end();) {
            if (m_goDown) {
                preJobVect.m_jobElemVecMgr_x.Unlock();
                log_DBG_m (dbg_NORM, "Shut down. Abort SendToPM.");
                return;
            }

            hsm_FileHeader &fh = ***iskip; // these *** are not HOTEL stars
            if (&fh == NULL) {
                skipped.erase(iskip++);
                continue; // probably moved to dirty list
            }
            //  try to lock FH by inode
            ivd_GenInode_t inode = fh.GetInode();
            if (!g_hsm_fhLock.CanLockByID(inode)) {
                // swap this element with last one
                hsm_FileHeader** temp = *iskip;
                *iskip = skipped.back();
                skipped.back() = temp;
                preJobVect.m_jobElemVecMgr_x.Unlock(); // to allow changes
                log_DBG_m(dbg_DETAIL, "Unlock job list and sleep a while. ");
                ivd_USleep(10);
                preJobVect.m_jobElemVecMgr_x.Lock();
                continue;  // can't use this FH already locked, so skip it
            }
            preJobVect.m_jobElemVecMgr_x.Unlock();

            ivd_FileID_t fileID = fh.GetFileID();

            if ( fileID == 0) {
                try { // TODO update fileID in HSMDB
                    fh.SetFileID((tFIDs)[FileIDidx++]);
                }
                catch (ivd_SysError &ise) {
                    // file is probably deletted in meantime
                    log_DBG_m(dbg_NORM, "Try 2. File is probably deleted. Simulate delete event.");
                    fio_Transaction &trans = *g_hsmDB_p->GetTransObj();
                    cmn_Path oldName(fh.GetFName());
                    fh.Event(fs_api::eDelFile, trans, &oldName);
                    g_hsmDB_p->ReleaseTransObj(trans);

                    skipped.erase(iskip++);
                    g_hsm_fhLock.UnLockByID(inode);
                    preJobVect.m_jobElemVecMgr_x.Lock();
                    continue;
                }
            }
            fh.MigPreJobToPM(seqTFileList[numfiles++]);

            if ( g_mig.GetAPIMode() == true ){
                g_mig.InsertJobInode(fh.GetInode(), a_preJobIdx);
            }


            skipped.erase(iskip++);
            g_hsm_fhLock.UnLockByID(inode);
            preJobVect.m_jobElemVecMgr_x.Lock();
        }
        preJobVect.m_jobElemVecMgr_x.Unlock();
    }

    if (numfiles == 0) {
        log_DBG_m(dbg_NORM, "Sequence is EMPTY, all files are changed in meantime. JobList "
                             + cmn_Num2Str(a_preJobIdx));
        g_hsm_preJobList_p->ReleaseJobIdx(a_preJobIdx);
        return;
    }

//    delete tFIDs;

    seqTFileList.length(numfiles);

    log_DBG_m(dbg_DETAIL, "  Sequence is prepared to call Migrate on PM of "
                        << seqTFileList.length() << " files");

    try {
        ipc_EXEC_m(
            i_JobID_t jobID;
            if (g_mig.GetAPIMode() == true) {
                jobID = m_iPM->Migrate(seqTFileList, 0, g_mig.m_majColId, g_mig.m_minColId, g_mig.m_flags);
                cmn_MutexLock l(g_jobList_x);
                g_mig.AddJob(jobID);
            }
            else {
                jobID = m_iPM->Migrate(seqTFileList, 0, m_majColId, m_minColId, m_flags);
                m_majColId = 0;
                m_minColId = 0;
                m_flags = 0;
            }

            log_DBG_m(dbg_NORM, "Job: " << jobID)

            g_hsm_preJobList_p->SetJobStatus(a_preJobIdx, hjs_WAITFORDOMIG);
            g_hsm_preJobList_p->SetMigJobID(a_preJobIdx, jobID, 0);

            log_DBG_m(dbg_DETAIL, "Called PM Migrate returned. Waiting for PM to call DoMigration.");
        );
    }
    catch (ivd_Exception &ie) {
        log_ERR_m("PM->Migrate() failed: " + ie.GetFriendly());
        ostringstream sstr;
        sstr << "Failed to start migration on " << g_cmn.prod.m_nameShort
             << "-PM (" << ie.GetFriendly() << ").";
        log_WriteEvent(evt_ERROR, sstr.str());
        g_hsm_preJobList_p->SetJobStatus(a_preJobIdx, hjs_MIGFAILED);
    }
    catch (...){
        log_ERR_m("PM->Migrate() failed: unknown exception");
        log_WriteEvent(evt_ERROR, "Failed to start migration on " + g_cmn.prod.m_nameShort + "-PM.");
        g_hsm_preJobList_p->SetJobStatus(a_preJobIdx, hjs_MIGFAILED);
    }
    return;
}

Here is the call graph for this function:

Here is the caller graph for this function:

bool hsm_FHmigc::ShouldMigrate (  )  [private]

Definition at line 471 of file hsm_FHmigc.cpp.

References cmn_Global::dbg, dbg_DETAIL, g_cmn, log_Debugger::GetLevel(), log_DBG_m, log_FUNC_m, hsm_FHlist::m_FHlist_x, m_firstAppendTime, m_lastAppendTime, m_maxMigSize, m_maxNumFiles, m_maxWaitTime, m_migDataSize, m_minMigSize, m_minNumFiles, m_minWaitTime, NULL, size, and cmn_Time::Time2hms().

Referenced by Append(), Migrate(), and Run().

                               {
    log_FUNC_m(ShouldMigrate);

    cmn_MutexLock l(m_FHlist_x);
    hsm_FH_p_l::size_type listsize = size();
    ivd_Time32_t now(time(NULL));
    bool allMinsFulfilled(   (m_migDataSize >= m_minMigSize) // size of files on list
                          && (listsize      >= m_minNumFiles) // number of files on list
                          && (now >= m_lastAppendTime + m_minWaitTime)); // age of the last file on the list

    bool anyMaxFulfilled(   (m_migDataSize >= m_maxMigSize) // size of files on list
                         || (listsize      >= m_maxNumFiles) // number of files on list
                         || (   (now >= m_firstAppendTime + m_maxWaitTime) // age of the first file on the list
                             && (listsize != 0)));

    if (g_cmn.dbg.GetLevel() == dbg_DETAIL) {
        cmn_Time firstAppendTime(m_firstAppendTime, 0);
        cmn_Time lastAppendTime(m_lastAppendTime, 0);
        log_DBG_m(dbg_DETAIL, "size of files=" << m_migDataSize
                            << ", list size=" << listsize
                            << ", first file added at " << firstAppendTime.Time2hms()
                            << ", last file added at " << lastAppendTime.Time2hms()
                            << ", allMinsFulfilled=" << boolalpha << allMinsFulfilled
                            << ", anyMaxFulfilled=" << boolalpha << anyMaxFulfilled);
    }
    return (allMinsFulfilled || anyMaxFulfilled);
}

Here is the call graph for this function:

Here is the caller graph for this function:

void hsm_FHmigc::Shutdown (  ) 

Definition at line 276 of file hsm_FHmigc.cpp.

References log_FUNC_m, m_goDown, and WakeUp().

Referenced by hsm_Containers::Stop().

                          {
    log_FUNC_m(Shutdown);
    m_goDown = true;
    WakeUp();
}

Here is the call graph for this function:

Here is the caller graph for this function:

UInt32_t hsm_FHmigc::TrigMigration ( void   ) 

Definition at line 148 of file hsm_FHmigc.cpp.

References cmn_Condition::Broadcast(), ClientConf_t::CfgRecallOnly, g_clientConf_p, g_hsm_preJobList_p, log_FUNC_m, hsm_FHlist::m_FHlist_x, m_forcedMigration, m_thread_c, m_thread_x, hsm_JobListMgr::MigrateFailedJob(), and size.

Referenced by i_HSM_i::MigrateByAPI(), and i_HSM_i::TrigMigration().

                                       {
    log_FUNC_m(TrigMigration);
// maybe will be used in future    g_hsm_activeList_p->OrphanedToMigList();
    UInt32_t count(0);
    if (!g_clientConf_p->CfgRecallOnly) {
        count += g_hsm_preJobList_p->MigrateFailedJob();
    }

    cmn_MutexLock l(m_thread_x);
    m_forcedMigration = true;
    m_thread_c.Broadcast();
    cmn_MutexLock listLock(m_FHlist_x);
    count += size();
    return count;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void hsm_FHmigc::WakeUp ( void   ) 

Definition at line 284 of file hsm_FHmigc.cpp.

References cmn_Condition::Broadcast(), log_FUNC_m, m_thread_c, and m_thread_x.

Referenced by Append(), and Shutdown().

Here is the call graph for this function:

Here is the caller graph for this function:


Member Data Documentation

Reimplemented from cmn_Thread.

Definition at line 186 of file hsm_FHmigc.h.

Definition at line 174 of file hsm_FHmigc.h.

Referenced by Append(), DumpStatus(), Run(), and ShouldMigrate().

Definition at line 183 of file hsm_FHmigc.h.

Referenced by SendToPM().

Definition at line 178 of file hsm_FHmigc.h.

Referenced by Run(), and TrigMigration().

bool hsm_FHmigc::m_goDown [private]

Definition at line 145 of file hsm_FHmigc.h.

Referenced by DumpList(), Migrate(), Run(), SendToPM(), and Shutdown().

i_PartitionManager_var hsm_FHmigc::m_iPM [private]

Definition at line 140 of file hsm_FHmigc.h.

Referenced by GetFileIDs(), hsm_FHmigc(), and SendToPM().

Definition at line 176 of file hsm_FHmigc.h.

Referenced by Append(), DumpStatus(), Run(), and ShouldMigrate().

Definition at line 181 of file hsm_FHmigc.h.

Referenced by Migrate(), and SendToPM().

If m_maxMigSize data is add to mig list then migrate it.

Definition at line 164 of file hsm_FHmigc.h.

Referenced by DumpStatus(), hsm_FHmigc(), Migrate(), Reconfigure(), and ShouldMigrate().

If m_maxNumFiles lay on list then migrate immediately.

Definition at line 162 of file hsm_FHmigc.h.

Referenced by DumpStatus(), hsm_FHmigc(), Migrate(), Reconfigure(), and ShouldMigrate().

If maxWaitTime passed from last migration then mig now.

Definition at line 160 of file hsm_FHmigc.h.

Referenced by DumpStatus(), hsm_FHmigc(), Reconfigure(), Run(), and ShouldMigrate().

Definition at line 172 of file hsm_FHmigc.h.

Referenced by Append(), GetFilesListSize(), Migrate(), Remove(), Run(), and ShouldMigrate().

Definition at line 166 of file hsm_FHmigc.h.

Definition at line 182 of file hsm_FHmigc.h.

Referenced by Migrate(), and SendToPM().

Amount of data that has to be achive to migrate in minWaitTime.

Definition at line 157 of file hsm_FHmigc.h.

Referenced by DumpStatus(), hsm_FHmigc(), Reconfigure(), and ShouldMigrate().

at least this number of files to be mig at minWaitTime

Definition at line 155 of file hsm_FHmigc.h.

Referenced by DumpStatus(), hsm_FHmigc(), Reconfigure(), and ShouldMigrate().

If nothing change in minWaitTime perion and see m_minNumMigFile, m_minMigSize.

Definition at line 153 of file hsm_FHmigc.h.

Referenced by DumpStatus(), hsm_FHmigc(), Reconfigure(), Run(), and ShouldMigrate().

bool& hsm_FHmigc::m_running [private]

outside variable that set when thread is down

Definition at line 143 of file hsm_FHmigc.h.

Referenced by Run().

Definition at line 149 of file hsm_FHmigc.h.

Referenced by Run(), TrigMigration(), and WakeUp().

Definition at line 148 of file hsm_FHmigc.h.

Referenced by Run(), TrigMigration(), and WakeUp().

thread wake up time

Definition at line 170 of file hsm_FHmigc.h.

Referenced by GetWakeUpTime(), and Run().


The documentation for this class was generated from the following files: