fsc_DataLMgr Class Reference
[File System Catalog]

#include <fsc_DataLMgr.h>

Collaboration diagram for fsc_DataLMgr:

[legend]

List of all members.

Detailed Description

<Description:> A more elaborate class description that describes all aspects of class usage etc.

Author:: Dejan Volk Lupo, HERMES SoftLab

See also:: <reference>

Definition at line 57 of file fsc_DataLMgr.h.


Public Member Functions
	fsc_DataLMgr (fio_Transaction *a_trans_p)
	Default constructor.
	fsc_DataLMgr (int a_cacheSize, fio_Transaction *a_trans_p)
void	FastInsert (df_SplitInfo &a_splitInfo)
	NOTE! used only for one thread at once.
void	FlushChangesAndCleanCache ()
	Every now and then is good to flush data, most common time is when number of different fileID are near to cache size.
void	Insert (df_SplitInfo &a_splitInfo)
	Insert a split into Data location tree.
void	Remove (const string &a_partName, ivd_MediaKey_t a_mediumKey, ivd_MedVolNum_t a_medVolNum)
	Remove all entries from particular volume.
bool	Remove (ivd_RecordIDX_t a_fileID, ivd_MediaKey_t a_mediumKey, ivd_MedVolNum_t a_medVolNum, UInt32_t a_blockOffs)
	Remove one particular entry.
fsc_DataL *	FindDataL (ivd_RecordIDX_t a_fileID)
void	GetBunchOfFileIDs (ivd_RecIdxVec_t &a_recIdxVec, UInt32_t a_num, fio_Transaction &a_trans)
void	GetCopiesPos (ivd_CopiesPos_v_t &a_copiesPos_v, ivd_FileID_t a_fileID, ivd_MigrationID_t &a_migID, ivd_FileSize_t a_fileSize)
void	CheckLastGen (string &a_status, i_FSCfileState_t &a_FSCFileState, const i_IvdfsFileInfo_t &a_ivdFSfileInfo, ivd_MigrationID_t &a_migID, UInt32_t a_numPools, const fsc_MediumStatus_v &a_ms)
void	DumpTree (ostream &os, ivd_FileID_t a_fileID)
bool	TreeWalk (ivd_FileID_t a_fileID, fsc_Collector &a_collector)
void	GetHistory (ivd_FileID_t a_fileID, fsc_GenIDsize_v &a_generations, ivd_MigrationID_t &a_nscMigID, string &a_path)
UInt32_t	GetNumOfUsedFileID ()
void	NumCopiesVsNumPoolsCheck (i_FSCNumCopVsNumPoolsCheckStatus_t &state, ivd_CopiesPos_v_t tmpCopiesPos_v, UInt32_t a_numPools, const fsc_MediumStatus_v &a_ms)
fsc_DataL *	GetDataL (ivd_RecordIDX_t a_fileID)
Private Attributes
fio_RelFileTrans &	m_relFile
	this is pointer to relative file "fsc_xxxx.lrf" object.
	log_CLASSID_m

Constructor & Destructor Documentation

fsc_DataLMgr::fsc_DataLMgr ( fio_Transaction * a_trans_p )

Default constructor.

construct for recall the FileID

Definition at line 46 of file fsc_DataLMgr.cpp.

References g_fsc_trans_p, and log_FUNC_m.

00047         :
00048         m_relFile(*g_dataLRF_p)
00049 {
00050     log_FUNC_m(fsc_DataLMgr);
00051     g_fsc_trans_p = a_trans_p;
00052     //empty
00053 }
//============================================================================//

fsc_DataLMgr::fsc_DataLMgr	(	int	a_cacheSize,
		fio_Transaction *	a_trans_p
	)

Definition at line 56 of file fsc_DataLMgr.cpp.

References g_fsc_trans_p, and log_FUNC_m.

00058         :
00059         m_relFile(*g_dataLRF_p)
00060 {    
00061     log_FUNC_m(fsc_DataLMgr);
00062     g_fsc_trans_p = a_trans_p;
00063 }
//============================================================================//

Member Function Documentation

void fsc_DataLMgr::FastInsert ( df_SplitInfo & a_splitInfo )

NOTE! used only for one thread at once.

Definition at line 372 of file fsc_DataLMgr.cpp.

References df_SplitInfo::fileID, g_DataLcache, stx_CacheArray< _MaxCacheSize >::Get(), GetDataL(), fsc_DataL::Insert(), stx_CacheArray< _MaxCacheSize >::Insert(), fsc_DataL::IsFree(), and NULL.

Referenced by Insert(), and fsc_FRIunpacker::ProcSplitInfo().

00372                                                        {    
00373 //    log_FUNC_m(FastInsert);
00374 
00375     ivd_RecordIDX_t fileID = a_splitInfo.fileID;
00376     fsc_fileIDLock l(fileID);
00377 
00378     fsc_DataL* data_p = static_cast<fsc_DataL*>(g_DataLcache.Get(fileID));
00379     if (data_p == NULL) {
00380         data_p = GetDataL(fileID);
00381         g_DataLcache.Insert(data_p);
00382     }
00383 
00384     if (data_p->IsFree()) { // element is freee to use
00385         new (data_p) fsc_DataL(a_splitInfo);
00386     }
00387     else {
00388         data_p->Insert(a_splitInfo);
00389     }
00390 }

Here is the call graph for this function:

Here is the caller graph for this function:

void fsc_DataLMgr::FlushChangesAndCleanCache ( )

Every now and then is good to flush data, most common time is when number of different fileID are near to cache size.

transaction is startted before flush and stopped soon after.

Definition at line 395 of file fsc_DataLMgr.cpp.

References stx_CacheArray< _MaxCacheSize >::CleanUpAndNotify(), fio_Transaction::EndTransaction(), g_DataLcache, g_fsc_trans_p, log_FUNC_m, and fio_Transaction::StartTransaction().

Referenced by i_FSC_i::FSCRecovery(), Insert(), i_FSC_i::InsertSplitFromFRIBlock(), fsc_FRIunpacker::ProcSplitInfo(), and Remove().

00395                                              {    
00396     log_FUNC_m(FlushChangesAndCleanCache);    
00397 
00398     g_fsc_trans_p->StartTransaction();
00399     g_DataLcache.CleanUpAndNotify();
00400     g_fsc_trans_p->EndTransaction();
00401 }

Here is the call graph for this function:

Here is the caller graph for this function:

void fsc_DataLMgr::Insert ( df_SplitInfo & a_splitInfo )

Insert a split into Data location tree.

Definition at line 66 of file fsc_DataLMgr.cpp.

References FastInsert(), FlushChangesAndCleanCache(), g_DataLcache, g_fscUpdate_x, log_FUNC_m, and stx_CacheArray< _MaxCacheSize >::Resize().

Referenced by fsc_VolumeCheck::Check().

00066                                                    {
00067     log_FUNC_m(Insert);
00068 
00069     cmn_MutexLock l(g_fscUpdate_x);
00070     // cache size will return at destructor
00071     g_DataLcache.Resize(1);
00072     FastInsert(a_splitInfo);
00073     FlushChangesAndCleanCache();
00074 }

Here is the call graph for this function:

Here is the caller graph for this function:

void fsc_DataLMgr::Remove	(	const string &	a_partName,
		ivd_MediaKey_t	a_mediumKey,
		ivd_MedVolNum_t	a_medVolNum
	)

Remove all entries from particular volume.

Definition at line 77 of file fsc_DataLMgr.cpp.

References ivd_FileLocationData_t::fileID, FlushChangesAndCleanCache(), g_DataLcache, g_fscUpdate_x, stx_CacheArray< _MaxCacheSize >::Get(), GetDataL(), fsc_DataCollectorStorage::GetNextRecord(), stx_CacheArray< _MaxCacheSize >::Insert(), cmn_Mutex::Lock(), log_FUNC_m, NULL, fsc_DataL::Remove(), stx_CacheArray< _MaxCacheSize >::Resize(), and cmn_Mutex::Unlock().

Referenced by fsc_VolumeCheck::Check(), fsc_CollectorDelete::ProcRecMedPosMgr(), i_FSC_i::ReleaseVolEntries(), Remove(), and fsc_VolumeCheck::RestDataHandle().

00079                                                            {
00080     log_FUNC_m(Remove);
00081     // cache size will return at destructor
00082     
00083     fsc_ReorgScanContentVolStorage volCont( 0,
00084                                             a_partName, 
00085                                             a_mediumKey, 
00086                                             a_medVolNum,
00087                                             false);
00088     //false == do not create files, they already exist
00089     
00090                                                     
00091     int transCount = 0;
00092     int cachSize = 1024;
00093     
00094     g_fscUpdate_x.Lock();
00095     g_DataLcache.Resize(cachSize);
00096     
00097     ivd_FileLocationData_t locData;
00098     ivd_RecordIDX_t lastFileID = 0;
00099     while (!volCont.GetNextRecord(locData)) {
00100 
00101         if (lastFileID == locData.fileID) {
00102             continue;
00103         }
00104         else {
00105             lastFileID = locData.fileID;
00106         }
00107         
00108         fsc_DataL* data_p;  
00109         {  // lock and auto unlock
00110             fsc_fileIDLock l(locData.fileID);
00111             data_p = static_cast<fsc_DataL*>(g_DataLcache.Get(locData.fileID));
00112             if (data_p == NULL) {
00113                 data_p = GetDataL(locData.fileID);
00114                 g_DataLcache.Insert(data_p);
00115             }
00116             data_p->Remove(a_mediumKey, a_medVolNum);
00117         }
00118         
00119         if (++transCount >= cachSize) {
00120             transCount = 0;
00121             FlushChangesAndCleanCache();
00122             g_fscUpdate_x.Unlock(); // let others to take a breath
00123             g_fscUpdate_x.Lock();
00124             g_DataLcache.Resize(cachSize);  // cache may resize
00125         }        
00126     };    
00127     FlushChangesAndCleanCache();
00128     g_fscUpdate_x.Unlock();
00129 }

Here is the call graph for this function:

Here is the caller graph for this function:

bool fsc_DataLMgr::Remove	(	ivd_RecordIDX_t	a_fileID,
		ivd_MediaKey_t	a_mediumKey,
		ivd_MedVolNum_t	a_medVolNum,
		UInt32_t	a_blockOffs
	)

Remove one particular entry.

Definition at line 132 of file fsc_DataLMgr.cpp.

References FlushChangesAndCleanCache(), g_DataLcache, g_fscUpdate_x, stx_CacheArray< _MaxCacheSize >::Get(), GetDataL(), stx_CacheArray< _MaxCacheSize >::Insert(), log_FUNC_A_m, NULL, fsc_DataL::Remove(), Remove(), and stx_CacheArray< _MaxCacheSize >::Resize().

00135                                                            {
00136     log_FUNC_A_m(Remove(), "a_fileID" << a_fileID );
00137     
00138     cmn_MutexLock l(g_fscUpdate_x);
00139     bool ret = false;
00140     
00141     g_DataLcache.Resize(1);
00142     
00143         fsc_DataL* data_p;  
00144         {  // lock and auto unlock
00145             fsc_fileIDLock l(a_fileID);
00146             data_p = static_cast<fsc_DataL*>(g_DataLcache.Get(a_fileID));
00147             if (data_p == NULL) {
00148                 data_p = GetDataL(a_fileID);
00149                 g_DataLcache.Insert(data_p);
00150             }
00151             ret = data_p->Remove(a_mediumKey, a_medVolNum, a_blockOffs);
00152         }
00153         
00154     FlushChangesAndCleanCache();
00155     return ret;
00156 }

Here is the call graph for this function:

fsc_DataL* fsc_DataLMgr::FindDataL ( ivd_RecordIDX_t a_fileID )

void fsc_DataLMgr::GetBunchOfFileIDs	(	ivd_RecIdxVec_t &	a_recIdxVec,
		UInt32_t	a_num,
		fio_Transaction &	a_trans
	)

Definition at line 159 of file fsc_DataLMgr.cpp.

References fio_Transaction::AbortTransaction(), fio_Transaction::EndTransaction(), fio_RelFileTrans::GetBunchOfVectorIDXes(), log_FUNC_m, m_relFile, fio_Transaction::StartTransaction(), and fio_RelFile::WriteRec().

Referenced by i_FSC_i::GetFileIDs().

00161                                                                {
00162     log_FUNC_m(GetBunchOfFileIDs);
00163 
00164     a_recIdxVec.clear();
00165     a_recIdxVec.resize(a_num);
00166 
00167     a_trans.StartTransaction();
00168     
00169     try {
00170         m_relFile.GetBunchOfVectorIDXes(a_recIdxVec, 
00171                                         1,  // vector size
00172                                         &a_trans);
00173 
00174         fsc_DataL_t dataL;  // some empty record to reset into DataL relative file
00175 
00176         // write empty records to all indeces
00177         for (ivd_RecIdxVec_i iter = a_recIdxVec.begin();
00178              iter != a_recIdxVec.end();
00179              iter++) {
00180             m_relFile.WriteRec(*iter, &dataL, 1, &a_trans);
00181         }
00182     }
00183     catch (...) {
00184         a_trans.AbortTransaction();
00185         throw;
00186     }
00187 
00188     a_trans.EndTransaction();
00189 }

Here is the call graph for this function:

Here is the caller graph for this function:

void fsc_DataLMgr::GetCopiesPos	(	ivd_CopiesPos_v_t &	a_copiesPos_v,
		ivd_FileID_t	a_fileID,
		ivd_MigrationID_t &	a_migID,
		ivd_FileSize_t	a_fileSize
	)

Definition at line 192 of file fsc_DataLMgr.cpp.

References fsc_DataL::GetCopiesPos(), GetDataL(), and log_FUNC_m.

Referenced by i_FSC_i::EfficientRecallGetLocations(), and i_FSC_i::GetCopiesPos().

00195                                                                {
00196     log_FUNC_m(GetCopiesPos);
00197 
00198     fsc_fileIDLock l(a_fileID);
00199 
00200     fsc_DataL *data_p = GetDataL(a_fileID);
00201     data_p->GetCopiesPos(a_copiesPos_v, a_migID, a_fileSize);
00202     delete data_p;   
00203 }

Here is the call graph for this function:

Here is the caller graph for this function:

void fsc_DataLMgr::CheckLastGen	(	string &	a_status,
		i_FSCfileState_t &	a_FSCFileState,
		const i_IvdfsFileInfo_t &	a_ivdFSfileInfo,
		ivd_MigrationID_t &	a_migID,
		UInt32_t	a_numPools,
		const fsc_MediumStatus_v &	a_ms
	)

Definition at line 255 of file fsc_DataLMgr.cpp.

References fsc_DataL::CheckLastGen(), i_IvdfsFileInfo_t::fileID, i_IvdfsFileInfo_t::fileSize, i_FSCfileState_t::fscNumCopVsNumPoolsCheckStatus, fsc_DataL::GetCopiesPos(), GetDataL(), ivd_BaseException::GetError(), ie_FSC_NOCOPIES, log_FUNC_m, and NumCopiesVsNumPoolsCheck().

Referenced by i_FSC_i::CheckWithIVDFS().

00260                                                                               {
00261     log_FUNC_m(CheckLastGen);
00262 
00263     ivd_FileID_t fileID = a_ivdFSfileInfo.fileID;
00264     fsc_fileIDLock l(fileID);
00265 
00266     fsc_DataL *data_p = GetDataL(fileID);
00267     data_p->CheckLastGen(a_status, a_FSCFileState, a_ivdFSfileInfo, a_migID);
00268     
00269     // Calling getCopiesPos on directory always rises exception because Dir has no DATA
00270     // IMPROVMENT: IvdFSfile info could be extended to contain type info (DIR, FILE,...)
00271     // can be done by extending the structure IvdfsFileInfo on HSM side see treewalk.cpp
00272     if (a_status.empty() ) {
00273         ivd_MigrationID_t tmpMigID = 0;
00274         ivd_CopiesPos_v_t tmpCopiesPos_v;
00275         try {
00276             data_p->GetCopiesPos(tmpCopiesPos_v, tmpMigID, a_ivdFSfileInfo.fileSize);
00277             NumCopiesVsNumPoolsCheck(
00278                 a_FSCFileState.fscNumCopVsNumPoolsCheckStatus, 
00279                 tmpCopiesPos_v, 
00280                 a_numPools, 
00281                 a_ms);
00282         }
00283         catch (ivd_Error &ie) {
00284             // if the error thrown is because file has no data just "ignore" it
00285             if (ie.GetError() != ie_FSC_NOCOPIES) {
00286                 throw;
00287             }
00288         }
00289     }
00290 
00291     delete data_p;   
00292 }

Here is the call graph for this function:

Here is the caller graph for this function:

void fsc_DataLMgr::DumpTree	(	ostream &	os,
		ivd_FileID_t	a_fileID
	)

Definition at line 305 of file fsc_DataLMgr.cpp.

References fsc_DataL::Dump(), g_DataLcache, stx_CacheArray< _MaxCacheSize >::Get(), GetDataL(), log_FUNC_m, and NULL.

Referenced by Interactive(), and main().

00305                                                               {
00306     log_FUNC_m(DumpTree);
00307 
00308     fsc_fileIDLock l(a_fileID);
00309 
00310     bool dumpCreateIt = false;
00311     fsc_DataL* data_p = static_cast<fsc_DataL*>(g_DataLcache.Get(a_fileID));
00312     if (data_p == NULL) {
00313         data_p = GetDataL(a_fileID);
00314         dumpCreateIt = true;
00315     }
00316 //    fsc_DataL *data_p = GetDataL(a_fileID);
00317     if (data_p == NULL) {
00318         os << "No data for fileID " << a_fileID << endl;
00319         return;
00320     }
00321     os << "================================" << endl;
00322     os << "Tree for fileID " << a_fileID << endl;
00323 
00324     data_p->Dump(os);
00325     if (dumpCreateIt) {
00326         delete data_p;   
00327     }
00328 }

Here is the call graph for this function:

Here is the caller graph for this function:

bool fsc_DataLMgr::TreeWalk	(	ivd_FileID_t	a_fileID,
		fsc_Collector &	a_collector
	)

void fsc_DataLMgr::GetHistory	(	ivd_FileID_t	a_fileID,
		fsc_GenIDsize_v &	a_generations,
		ivd_MigrationID_t &	a_nscMigID,
		string &	a_path
	)

Definition at line 354 of file fsc_DataLMgr.cpp.

References GetDataL(), fsc_DataL::GetFullPath(), log_FUNC_m, and fsc_DataL::TreeWalk().

Referenced by i_FSC_i::GetFileHistory().

00357                                                          {
00358     log_FUNC_m(GetHistory);
00359     
00360     fsc_ColGeneration colGen(a_fileID, a_generations);
00361 
00362     fsc_DataL *data_p = GetDataL(a_fileID);
00363 
00364     fsc_fileIDLock l(a_fileID);
00365 
00366     data_p->TreeWalk(colGen);
00367     data_p->GetFullPath(a_nscMigID, a_path);
00368     delete data_p;
00369 }

Here is the call graph for this function:

Here is the caller graph for this function:

UInt32_t fsc_DataLMgr::GetNumOfUsedFileID ( ) [inline]

Definition at line 124 of file fsc_DataLMgr.h.

References fio_RelFile::GetNumOfRecord().

00124 { return m_relFile.GetNumOfRecord(); };

Here is the call graph for this function:

void fsc_DataLMgr::NumCopiesVsNumPoolsCheck	(	i_FSCNumCopVsNumPoolsCheckStatus_t &	state,
		ivd_CopiesPos_v_t	tmpCopiesPos_v,
		UInt32_t	a_numPools,
		const fsc_MediumStatus_v &	a_ms
	)

Definition at line 206 of file fsc_DataLMgr.cpp.

References i_FSCNumCopVsNumPoolsCheckStatus_t::changed, i_FSCNumCopVsNumPoolsCheckStatus_t::copiesExpected, i_FSCNumCopVsNumPoolsCheckStatus_t::copiesFound, log_FUNC_m, i_FSCNumCopVsNumPoolsCheckStatus_t::splitInWrongPool, and i_FSCNumCopVsNumPoolsCheckStatus_t::tooMuchCopiesPerPool.

Referenced by CheckLastGen().

00209                                                                                        {
00210     log_FUNC_m(NumCopiesVsNumPoolsCheck);
00211 
00212     vector <UInt32_t> copyCounter_v; //vector index = pool number, vector contents = copies per pool
00213     UInt32_t numOfCopies(0);  //only first copy from a pool is counted
00214 
00215     for (UInt32_t i = 0; i < tmpCopiesPos_v.size(); ++i) {
00216         UInt32_t poolKey(0);
00217         for (UInt32_t j = 0; j < tmpCopiesPos_v[i].size(); ++j) {
00218             UInt32_t msPoolKey = a_ms[tmpCopiesPos_v[i][j].mediaKey].mediaPoolKey;
00219             if (poolKey == 0) {
00220                 poolKey = msPoolKey;
00221             }
00222             else if (poolKey != msPoolKey && !state.splitInWrongPool) {
00223                 state.splitInWrongPool = true;
00224                 state.changed = true;
00225            }
00226         }
00227 
00228         UInt32_t copyCounter_v_Size;
00229 
00230         while(poolKey + 1 > copyCounter_v.size()) {
00231             copyCounter_v_Size = copyCounter_v.size();
00232             copyCounter_v.resize(copyCounter_v_Size + 1);
00233             copyCounter_v[copyCounter_v_Size] = 0;
00234         }
00235         ++copyCounter_v[poolKey];
00236 
00237         //count only first copy in pool, other copy is skipped from count
00238         if (copyCounter_v[poolKey] == 1) {
00239             ++numOfCopies;
00240         }
00241         else if (copyCounter_v[poolKey] > 2) {
00242             state.tooMuchCopiesPerPool = true;
00243             state.changed = true;
00244         }
00245     }
00246 
00247     if (numOfCopies != a_numPools) {
00248         state.copiesFound = numOfCopies;
00249         state.copiesExpected = a_numPools;
00250         state.changed = true;
00251     }
00252 }

Here is the caller graph for this function:

fsc_DataL* fsc_DataLMgr::GetDataL ( ivd_RecordIDX_t a_fileID ) [inline]

Definition at line 132 of file fsc_DataLMgr.h.

Referenced by CheckLastGen(), DumpTree(), FastInsert(), GetCopiesPos(), GetHistory(), and Remove().

00132                                                   {
00133         return new fsc_DataL(a_fileID);
00134     };

Here is the caller graph for this function:

Member Data Documentation

fio_RelFileTrans& fsc_DataLMgr::m_relFile [private]

this is pointer to relative file "fsc_xxxx.lrf" object.

It's not need to delete it in destructor

Definition at line 69 of file fsc_DataLMgr.h.

Referenced by GetBunchOfFileIDs().

fsc_DataLMgr::log_CLASSID_m [private]

Definition at line 71 of file fsc_DataLMgr.h.

The documentation for this class was generated from the following files:

fsc_DataLMgr Class Reference [File System Catalog]

Detailed Description

Public Member Functions

Private Attributes

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation

fsc_DataLMgr Class Reference
[File System Catalog]