xuemingqiang/mongo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fff9db5a8d5dab5cb2ec7dcd8d785859bb62f8ea
mongo/client/distlock_test.cpp

439 lines
15 KiB
C++
Raw Normal View History

distlock cleaning
2010-07-12 17:53:02 -04:00
// distlock_test.h
new file
2010-07-12 17:50:26 -04:00
/* Copyright 2009 10gen Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
#include <iostream>
new file
2010-07-12 17:50:26 -04:00
#include "../pch.h"
#include "dbclient.h"
#include "distlock.h"
#include "../db/commands.h"
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
#include "../util/bson_util.h"
Fix for distlock test - old version of boost::random on MSVC
2011-03-02 11:56:47 -05:00
// Modify some config options for the RNG, since they cause MSVC to fail
#include <boost/config.hpp>
#if defined(BOOST_MSVC) && defined(BOOST_NO_MEMBER_TEMPLATE_FRIENDS)
#undef BOOST_NO_MEMBER_TEMPLATE_FRIENDS
#define BOOST_RNG_HACK
#endif
// Well, sort-of cross-platform RNG
Cross platform RNG
2011-03-01 16:04:57 -05:00
#include <boost/random/mersenne_twister.hpp>
Fix for distlock test - old version of boost::random on MSVC
2011-03-02 11:56:47 -05:00
#ifdef BOOST_RNG_HACK
#define BOOST_NO_MEMBER_TEMPLATE_FRIENDS
#undef BOOST_RNG_HACK
#endif
Cross platform RNG
2011-03-01 16:04:57 -05:00
#include <boost/random/uniform_int.hpp>
#include <boost/random/variate_generator.hpp>
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
// TODO: Make a method in BSONObj if useful, don't modify for now
#define string_field(obj, name, def) ( obj.hasField(name) ? obj[name].String() : def )
#define number_field(obj, name, def) ( obj.hasField(name) ? obj[name].Number() : def )
new file
2010-07-12 17:50:26 -04:00
namespace mongo {
ran astyle SERVER-2304
2011-01-04 00:40:41 -05:00
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
class TestDistLockWithSync: public Command {
new file
2010-07-12 17:50:26 -04:00
public:
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
TestDistLockWithSync() :
Command("_testDistLockWithSyncCluster") {
}
virtual void help(stringstream& help) const {
new file
2010-07-12 17:50:26 -04:00
help << "should not be calling this directly" << endl;
}
ran astyle SERVER-2304
2011-01-04 00:40:41 -05:00
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
virtual bool slaveOk() const {
return false;
}
virtual bool adminOnly() const {
return true;
}
virtual LockType locktype() const {
return NONE;
}
new file
2010-07-12 17:50:26 -04:00
ran astyle SERVER-2304
2011-01-04 00:40:41 -05:00
static void runThread() {
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
while (keepGoing) {
Re-entrant distributed locks with ping unlock retries -- SERVER-3024 Retries unlock when process is still up but config server has gone down, in case of errors during locking and unlocking. Also allows testing of locks already held.
2011-05-03 11:21:49 -04:00
if (current->lock_try( "test" )) {
better distlock test
2011-02-14 16:18:52 -05:00
count++;
better test
2011-02-14 12:39:53 -05:00
int before = count;
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
sleepmillis(3);
better test
2011-02-14 12:39:53 -05:00
int after = count;
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
if (after != before) {
error() << " before: " << before << " after: " << after
<< endl;
better test
2011-02-14 12:39:53 -05:00
}
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
new file
2010-07-12 17:50:26 -04:00
current->unlock();
}
}
}
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
change command api to allow command options SERVER-3405
2011-07-18 14:25:18 -04:00
bool run(const string&, BSONObj& cmdObj, int, string& errmsg,
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
BSONObjBuilder& result, bool) {
better test
2011-02-14 12:39:53 -05:00
Timer t;
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
DistributedLock lk(ConnectionString(cmdObj["host"].String(),
remove legacy logic, refactor to consts SERVER-3047
2011-05-09 15:30:17 -04:00
ConnectionString::SYNC), "testdistlockwithsync", 0, 0);
new file
2010-07-12 17:50:26 -04:00
current = &lk;
count = 0;
gotit = 0;
better test
2011-02-14 12:39:53 -05:00
errors = 0;
better distlock test
2011-02-14 16:18:52 -05:00
keepGoing = true;
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
new file
2010-07-12 17:50:26 -04:00
vector<shared_ptr<boost::thread> > l;
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
for (int i = 0; i < 4; i++) {
l.push_back(
shared_ptr<boost::thread> (new boost::thread(runThread)));
new file
2010-07-12 17:50:26 -04:00
}
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
better distlock test
2011-02-14 16:18:52 -05:00
int secs = 10;
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
if (cmdObj["secs"].isNumber())
better distlock test
2011-02-14 16:18:52 -05:00
secs = cmdObj["secs"].numberInt();
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
sleepsecs(secs);
better distlock test
2011-02-14 16:18:52 -05:00
keepGoing = false;
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
for (unsigned i = 0; i < l.size(); i++)
new file
2010-07-12 17:50:26 -04:00
l[i]->join();
better test
2011-02-14 12:39:53 -05:00
current = 0;
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
result.append("count", count);
result.append("gotit", gotit);
result.append("errors", errors);
result.append("timeMS", t.millis());
ran astyle SERVER-2304
2011-01-04 00:40:41 -05:00
better distlock test
2011-02-14 16:18:52 -05:00
return errors == 0;
better test
2011-02-14 12:39:53 -05:00
}
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
better test
2011-02-14 12:39:53 -05:00
// variables for test
new file
2010-07-12 17:50:26 -04:00
static DistributedLock * current;
static int gotit;
better test
2011-02-14 12:39:53 -05:00
static int errors;
better distlock test
2011-02-14 16:18:52 -05:00
static AtomicUInt count;
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
better distlock test
2011-02-14 16:18:52 -05:00
static bool keepGoing;
new file
2010-07-12 17:50:26 -04:00
} testDistLockWithSyncCmd;
DistributedLock * TestDistLockWithSync::current;
better distlock test
2011-02-14 16:18:52 -05:00
AtomicUInt TestDistLockWithSync::count;
new file
2010-07-12 17:50:26 -04:00
int TestDistLockWithSync::gotit;
better test
2011-02-14 12:39:53 -05:00
int TestDistLockWithSync::errors;
better distlock test
2011-02-14 16:18:52 -05:00
bool TestDistLockWithSync::keepGoing;
new file
2010-07-12 17:50:26 -04:00
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
class TestDistLockWithSkew: public Command {
public:
more informative dist lock logging
2011-03-04 16:54:37 -05:00
static const int logLvl = 1;
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
TestDistLockWithSkew() :
Command("_testDistLockWithSkew") {
}
virtual void help(stringstream& help) const {
help << "should not be calling this directly" << endl;
}
virtual bool slaveOk() const {
return false;
}
virtual bool adminOnly() const {
return true;
}
virtual LockType locktype() const {
return NONE;
}
void runThread(ConnectionString& hostConn, unsigned threadId, unsigned seed,
BSONObj& cmdObj, BSONObjBuilder& result) {
stringstream ss;
ss << "thread-" << threadId;
setThreadName(ss.str().c_str());
// Lock name
string lockName = string_field(cmdObj, "lockName", this->name + "_lock");
// Range of clock skew in diff threads
int skewRange = (int) number_field(cmdObj, "skewRange", 1);
// How long to wait with the lock
int threadWait = (int) number_field(cmdObj, "threadWait", 30);
if(threadWait <= 0) threadWait = 1;
// Max amount of time (ms) a thread waits before checking the lock again
int threadSleep = (int) number_field(cmdObj, "threadSleep", 30);
if(threadSleep <= 0) threadSleep = 1;
// How long until the lock is forced in ms, only compared locally
Misspelling
2011-03-01 12:28:54 -08:00
unsigned long long takeoverMS = (unsigned long long) number_field(cmdObj, "takeoverMS", 0);
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
// Whether or not we should hang some threads
int hangThreads = (int) number_field(cmdObj, "hangThreads", 0);
Cross platform RNG
2011-03-01 16:04:57 -05:00
Fix for distlock test - old version of boost::random on MSVC
2011-03-02 11:56:47 -05:00
boost::mt19937 gen((boost::mt19937::result_type) seed);
Cross platform RNG
2011-03-01 16:04:57 -05:00
boost::variate_generator<boost::mt19937&, boost::uniform_int<> > randomSkew(gen, boost::uniform_int<>(0, skewRange));
boost::variate_generator<boost::mt19937&, boost::uniform_int<> > randomWait(gen, boost::uniform_int<>(1, threadWait));
boost::variate_generator<boost::mt19937&, boost::uniform_int<> > randomSleep(gen, boost::uniform_int<>(1, threadSleep));
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
int skew = 0;
if (!lock.get()) {
// Pick a skew, but the first two threads skew the whole range
if(threadId == 0)
skew = -skewRange / 2;
else if(threadId == 1)
skew = skewRange / 2;
Cross platform RNG
2011-03-01 16:04:57 -05:00
else skew = randomSkew() - (skewRange / 2);
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
// Skew this thread
jsTimeVirtualThreadSkew( skew );
log() << "Initializing lock with skew of " << skew << " for thread " << threadId << endl;
remove legacy logic, refactor to consts SERVER-3047
2011-05-09 15:30:17 -04:00
lock.reset(new DistributedLock(hostConn, lockName, takeoverMS, true ));
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
log() << "Skewed time " << jsTime() << " for thread " << threadId << endl
<< " max wait (with lock: " << threadWait << ", after lock: " << threadSleep << ")" << endl
remove legacy logic, refactor to consts SERVER-3047
2011-05-09 15:30:17 -04:00
<< " takeover in " << takeoverMS << "(ms remote)" << endl;
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
}
DistributedLock* myLock = lock.get();
bool errors = false;
Re-entrant distributed locks with ping unlock retries -- SERVER-3024 Retries unlock when process is still up but config server has gone down, in case of errors during locking and unlocking. Also allows testing of locks already held.
2011-05-03 11:21:49 -04:00
BSONObj lockObj;
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
while (keepGoing) {
more informative dist lock logging
2011-03-04 16:54:37 -05:00
try {
Re-entrant distributed locks with ping unlock retries -- SERVER-3024 Retries unlock when process is still up but config server has gone down, in case of errors during locking and unlocking. Also allows testing of locks already held.
2011-05-03 11:21:49 -04:00
if (myLock->lock_try("Testing distributed lock with skew.", false, &lockObj )) {
more informative dist lock logging
2011-03-04 16:54:37 -05:00
Re-entrant distributed locks with ping unlock retries -- SERVER-3024 Retries unlock when process is still up but config server has gone down, in case of errors during locking and unlocking. Also allows testing of locks already held.
2011-05-03 11:21:49 -04:00
log() << "**** Locked for thread " << threadId << " with ts " << lockObj["ts"] << endl;
remove legacy logic, refactor to consts SERVER-3047
2011-05-09 15:30:17 -04:00
if( count % 2 == 1 && ! myLock->lock_try( "Testing lock re-entry.", true ) ) {
errors = true;
log() << "**** !Could not re-enter lock already held" << endl;
break;
}
Re-entrant distributed locks with ping unlock retries -- SERVER-3024 Retries unlock when process is still up but config server has gone down, in case of errors during locking and unlocking. Also allows testing of locks already held.
2011-05-03 11:21:49 -04:00
remove legacy logic, refactor to consts SERVER-3047
2011-05-09 15:30:17 -04:00
if( count % 3 == 1 && myLock->lock_try( "Testing lock non-re-entry.", false ) ) {
errors = true;
log() << "**** !Invalid lock re-entry" << endl;
break;
Re-entrant distributed locks with ping unlock retries -- SERVER-3024 Retries unlock when process is still up but config server has gone down, in case of errors during locking and unlocking. Also allows testing of locks already held.
2011-05-03 11:21:49 -04:00
}
more informative dist lock logging
2011-03-04 16:54:37 -05:00
count++;
int before = count;
int sleep = randomWait();
sleepmillis(sleep);
int after = count;
if(after != before) {
errors = true;
log() << "**** !Bad increment while sleeping with lock for: " << sleep << "ms" << endl;
break;
}
// Unlock only half the time...
if(hangThreads == 0 || threadId % hangThreads != 0) {
Re-entrant distributed locks with ping unlock retries -- SERVER-3024 Retries unlock when process is still up but config server has gone down, in case of errors during locking and unlocking. Also allows testing of locks already held.
2011-05-03 11:21:49 -04:00
log() << "**** Unlocking for thread " << threadId << " with ts " << lockObj["ts"] << endl;
myLock->unlock( &lockObj );
more informative dist lock logging
2011-03-04 16:54:37 -05:00
}
else {
log() << "**** Not unlocking for thread " << threadId << endl;
DistributedLock::killPinger( *myLock );
// We're simulating a crashed process...
break;
}
}
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
}
catch( LockException& e ) {
conversion of unneeded caused_by macros to methods
2011-03-31 11:32:47 -04:00
log() << "*** !Could not try distributed lock." << causedBy( e ) << endl;
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
break;
}
Cross platform RNG
2011-03-01 16:04:57 -05:00
sleepmillis(randomSleep());
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
}
result << "errors" << errors
<< "skew" << skew
remove legacy logic, refactor to consts SERVER-3047
2011-05-09 15:30:17 -04:00
<< "takeover" << (long long) takeoverMS
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
<< "localTimeout" << (takeoverMS > 0);
}
void test(ConnectionString& hostConn, string& lockName, unsigned seed) {
return;
}
change command api to allow command options SERVER-3405
2011-07-18 14:25:18 -04:00
bool run(const string&, BSONObj& cmdObj, int, string& errmsg,
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
BSONObjBuilder& result, bool) {
Timer t;
ConnectionString hostConn(cmdObj["host"].String(),
ConnectionString::SYNC);
unsigned seed = (unsigned) number_field(cmdObj, "seed", 0);
int numThreads = (int) number_field(cmdObj, "numThreads", 4);
int wait = (int) number_field(cmdObj, "wait", 10000);
log() << "Starting " << this->name << " with -" << endl
<< " seed: " << seed << endl
<< " numThreads: " << numThreads << endl
<< " total wait: " << wait << endl << endl;
// Skew host clocks if needed
try {
skewClocks( hostConn, cmdObj );
}
catch( DBException e ) {
conversion of unneeded caused_by macros to methods
2011-03-31 11:32:47 -04:00
errmsg = str::stream() << "Clocks could not be skewed." << causedBy( e );
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
return false;
}
count = 0;
keepGoing = true;
vector<shared_ptr<boost::thread> > threads;
vector<shared_ptr<BSONObjBuilder> > results;
for (int i = 0; i < numThreads; i++) {
results.push_back(shared_ptr<BSONObjBuilder> (new BSONObjBuilder()));
threads.push_back(shared_ptr<boost::thread> (new boost::thread(
boost::bind(&TestDistLockWithSkew::runThread, this,
hostConn, (unsigned) i, seed + i, boost::ref(cmdObj),
boost::ref(*(results[i].get()))))));
}
sleepsecs(wait / 1000);
keepGoing = false;
bool errors = false;
for (unsigned i = 0; i < threads.size(); i++) {
threads[i]->join();
errors = errors || results[i].get()->obj()["errors"].Bool();
}
result.append("count", count);
result.append("errors", errors);
result.append("timeMS", t.millis());
return !errors;
}
/**
* Skews the clocks of a remote cluster by a particular amount, specified by
* the "skewHosts" element in a BSONObj.
*/
static void skewClocks( ConnectionString& cluster, BSONObj& cmdObj ) {
vector<long long> skew;
if(cmdObj.hasField("skewHosts")) {
bsonArrToNumVector<long long>(cmdObj["skewHosts"], skew);
}
more informative dist lock logging
2011-03-04 16:54:37 -05:00
else {
log( logLvl ) << "No host clocks to skew." << endl;
return;
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
}
more informative dist lock logging
2011-03-04 16:54:37 -05:00
log( logLvl ) << "Skewing clocks of hosts " << cluster << endl;
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
unsigned s = 0;
for(vector<long long>::iterator i = skew.begin(); i != skew.end(); ++i,s++) {
ConnectionString server( cluster.getServers()[s] );
ScopedDbConnection conn( server );
BSONObj result;
try {
bool success = conn->runCommand( string("admin"), BSON( "_skewClockCommand" << 1 << "skew" << *i ), result );
conversion of unneeded caused_by macros to methods
2011-03-31 11:32:47 -04:00
uassert(13678, str::stream() << "Could not communicate with server " << server.toString() << " in cluster " << cluster.toString() << " to change skew by " << *i, success );
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
more informative dist lock logging
2011-03-04 16:54:37 -05:00
log( logLvl + 1 ) << " Skewed host " << server << " clock by " << *i << endl;
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
}
catch(...) {
conn.done();
throw;
}
conn.done();
}
}
// variables for test
thread_specific_ptr<DistributedLock> lock;
AtomicUInt count;
bool keepGoing;
} testDistLockWithSkewCmd;
/**
* Utility command to virtually skew the clock of a mongo server a particular amount.
* This skews the clock globally, per-thread skew is also possible.
*/
class SkewClockCommand: public Command {
public:
SkewClockCommand() :
Command("_skewClockCommand") {
}
virtual void help(stringstream& help) const {
help << "should not be calling this directly" << endl;
}
virtual bool slaveOk() const {
return false;
}
virtual bool adminOnly() const {
return true;
}
virtual LockType locktype() const {
return NONE;
}
change command api to allow command options SERVER-3405
2011-07-18 14:25:18 -04:00
bool run(const string&, BSONObj& cmdObj, int, string& errmsg,
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
BSONObjBuilder& result, bool) {
long long skew = (long long) number_field(cmdObj, "skew", 0);
log() << "Adjusting jsTime() clock skew to " << skew << endl;
jsTimeVirtualSkew( skew );
log() << "JSTime adjusted, now is " << jsTime() << endl;
return true;
}
} testSkewClockCommand;
new file
2010-07-12 17:50:26 -04:00
}
Fixes for distributed lock handling time skew. In general, the current distributed lock is susceptible to time skew in config servers and mongos clients. This update uses an algorithm that checks for time skew, as well as providing some more robust tests to make sure a distributed lock cannot be forced prematurely. Some documentation on the website, currently hidden from non-10gen, gives more info.
2011-02-15 18:08:59 -05:00
Reference in New Issue Copy Permalink