Merge pull request #1512 from wiredtiger/2.5.0-release

Candidate for 2.5.0 release changelog

NEWS

@@ -1,3 +1,122 @@
+WiredTiger release 2.5.0, 2014-12-24
+------------------------------------
+
+The WiredTiger 2.5.0 release contains significant new features, API changes
+and many bug fixes.
+
+Now that WiredTiger is part of MongoDB, we are tracking issues related to
+MongoDB usage of WiredTiger in the MongoDB JIRA system.  Some entries in
+the changelog now reference JIRA tickets that can be found at:
+
+  http://jira.mongodb.org
+
+New features and API changes:
+
+* Add support for storing large values on-page in a btree rather than in
+  an overflow item.  This is useful for workloads that want to keep large
+  items in cache - since WiredTiger overflow items are never cached.
+  It is configured via new `leaf_value_max` configuration setting.  This
+  enhancement led to the deprecation of internal_item_max and leaf_item_max
+  configuration settings, see upgrading documentation for further
+  information. [#1282]
+
+* Add support for compressing log files.  When configured each compressable
+  log record will be compressed.  This is configure with the
+  `log=(compressor=X)` configuration setting.  See upgrading documentation
+  for further information. [#1359]
+
+* No longer return EBUSY when opening a bulk cursor, verifying or salvaging
+  a database if a checkpoint is currently running.  This allows
+  applications to do these exclusive operations without shutting down the
+  checkpoint server thread.  [#1397] [#1404] SERVER-16236 SERVER-16457
+
+* Add support for immutable indexes. [#1344]
+
+* Added several new statistics, improved accuracy for some statistics
+  tracking and simplified mechanism for querying a particular statistic.
+  [#1505]
+
+* Have the eviction server write out unnecessary pages prior to the cache
+  reaching the configured eviction trigger size.  This can reduce the
+  amount of eviction application threads do when configured with a large
+  cache.
+
+* Several enhancements to managing how long we keep files open
+
+* Revert a change in the 2.4.1 release that caused the WT_ROLLBACK (and
+  deprecated WT_DEADLOCK) error return to map to different numeric values.
+  Applications should ensure they are compiling against the same version
+  of the wiredtiger.h header file as the library they link against,
+  otherwise odd behavior will be experienced.
+
+* Support setting configuration strings to "none" as being equivalent to an
+  empty string in most cases. [#1417]
+
+* Enhance the hot backup implementation to allow recovery to be run between
+  incremental backups. [#1183]
+
+Other significant changes:
+
+* Improve performance of cursor open when there are many tables in a database.
+  [#1391] [#1443]
+
+* Reduce the impact checkpoints have on concurrent operations. This was
+  done by changing how we lock tables. [#1391] [#1392]
+
+* Improve performance when scanning a table that has many deleted items.
+  SERVER-16247
+
+* Fix a bug in checkpoint, where the metadata (turtle) file wasn't being
+  synced on checkpoint. [#1383]
+
+* Fix a bug where WiredTiger could accumulate memory during page splits and
+  never free it.  SERVER-16546
+
+* Many enhancements and bug fixes for Windows.
+
+* Fix a bug where a custom extractor terminate was being called twice.
+  [#1503]
+
+* Fix a bug where a race between closing a handle and checkpointing could
+  lead to errors. [#1495] [#1497]
+
+* Validate the block header checksum before we clear it - if the checksum
+  field had been corrupted, we didn't notice.  SERVER-16457
+
+* Fix a bug in write conflict detection.  Cursors configured with
+  no-overwrite could sometimes not see update conflicts for deleted
+  records.  SERVER-16351
+
+* Several bug fixes and performance improvements in LSM including:
+ - Add support for custom collators in LSM trees. [#1361]
+ - Fix a bug in LSM search_near, where it returned a deleted item.
+   BF-694, BF-700
+ - Improve background maintenance operations so that the cache does
+   not get full unnecessarily.
+ - Fix a bug that could lead to updates being written into old chunks.
+   [#1432] [#1418]
+ - Fix a bug in background merge that could skip updates.  SERVER-16123
+
+* Fix a bug when maintaining the cache, that could cause checkpoints to
+  skip writing an update that should have been included. [#1419] SERVER-16336
+
+* Fix a bug in WT_SESSION::drop, where failures generated error output even
+  when force was specified. [#1436]
+
+* Fix a bug in WiredTiger integer packing code when figuring out how much
+  space is required for a value (it can shrink as numbers grow). SERVER-16118
+
+* Several enhancements to the wtstats.py tool that generates graphs from
+  standard WiredTiger statistics logs. [#1365]
+
+* Fix a bug on OS X where fsync isn't sufficient to flush a file, use
+  fcntl(F_FULLFSYNC) instead.
+
+* Work around a bug in clang 3.5.0 compare and swap primitive. The
+  __sync_bool_compare_and_swap version of the API in clang produces bad
+  code for us with -O3 optimization enabled.
+
+
 WiredTiger release 2.4.1, 2014-11-06
 ------------------------------------
 

README

@@ -1,6 +1,6 @@
-WiredTiger 2.4.2: (November  6, 2014)
+WiredTiger 2.5.0: (December 24, 2014)
 
-This is version 2.4.2 of WiredTiger.
+This is version 2.5.0 of WiredTiger.
 
 WiredTiger release packages and documentation can be found at:
 
@@ -9,7 +9,7 @@ WiredTiger release packages and documentation can be found at:
 Information on configuring, building and installing WiredTiger can be
 found at:
 
-	http://source.wiredtiger.com/2.4.2/install.html
+	http://source.wiredtiger.com/2.5.0/install.html
 
 WiredTiger licensing information can be found at:
 

RELEASE_INFO

@@ -1,6 +1,6 @@
 WIREDTIGER_VERSION_MAJOR=2
-WIREDTIGER_VERSION_MINOR=4
-WIREDTIGER_VERSION_PATCH=2
+WIREDTIGER_VERSION_MINOR=5
+WIREDTIGER_VERSION_PATCH=0
 WIREDTIGER_VERSION="$WIREDTIGER_VERSION_MAJOR.$WIREDTIGER_VERSION_MINOR.$WIREDTIGER_VERSION_PATCH"
 
 WIREDTIGER_RELEASE_DATE=`date "+%B %e, %Y"`

build_posix/aclocal/version-set.m4

@@ -1,14 +1,14 @@
 dnl build by dist/s_version
 
 VERSION_MAJOR=2
-VERSION_MINOR=4
-VERSION_PATCH=2
-VERSION_STRING='"WiredTiger 2.4.2: (November  6, 2014)"'
+VERSION_MINOR=5
+VERSION_PATCH=0
+VERSION_STRING='"WiredTiger 2.5.0: (December 24, 2014)"'
 
 AC_SUBST(VERSION_MAJOR)
 AC_SUBST(VERSION_MINOR)
 AC_SUBST(VERSION_PATCH)
 AC_SUBST(VERSION_STRING)
 
-VERSION_NOPATCH=2.4
+VERSION_NOPATCH=2.5
 AC_SUBST(VERSION_NOPATCH)

build_posix/aclocal/version.m4

@@ -1,2 +1,2 @@
 dnl WiredTiger product version for AC_INIT.  Maintained by dist/s_version
-2.4.2
+2.5.0

dist/log.py

@@ -250,6 +250,7 @@ __wt_logop_%(name)s_unpack(
     'fmt' : op_pack_fmt(optype)
 })
 
+    last_field = optype.fields[-1]
     tfile.write('''
 int
 __wt_logop_%(name)s_print(
@@ -276,7 +277,10 @@ __wt_logop_%(name)s_print(
     'print_args' : '\n\t'.join(
         '%sfprintf(out, "    \\"%s\\": \\"%s\\",\\n",%s);' %
         (printf_setup(f), f[1], printf_fmt(f), printf_arg(f))
-        for f in optype.fields),
+        for f in optype.fields[:-1]) + str(
+        '\n\t%sfprintf(out, "    \\"%s\\": \\"%s\\"",%s);' %
+        (printf_setup(last_field), last_field[1],
+            printf_fmt(last_field), printf_arg(last_field))),
 })
 
 # Emit the printlog entry point

dist/package/wiredtiger.spec

@@ -1,5 +1,5 @@
 Name:		wiredtiger
-Version:	2.4.2
+Version:	2.5.0
 Release:	1%{?dist}
 Summary:	WiredTiger data storage engine
 

src/cursor/cur_log.c

@@ -12,12 +12,13 @@
  *	Callback function from log_scan to get a log record.
  */
 static int
-__curlog_logrec(
-    WT_SESSION_IMPL *session, WT_ITEM *logrec, WT_LSN *lsnp, void *cookie)
+__curlog_logrec(WT_SESSION_IMPL *session,
+    WT_ITEM *logrec, WT_LSN *lsnp, void *cookie, int firstrecord)
 {
 	WT_CURSOR_LOG *cl;
 
 	cl = cookie;
+	WT_UNUSED(firstrecord);
 
 	/* Set up the LSNs and take a copy of the log record for the cursor. */
 	*cl->cur_lsn = *lsnp;

src/docs/top/main.dox

@@ -6,12 +6,12 @@ WiredTiger is an high performance, scalable, production quality, NoSQL,
 @section releases Releases
 
 <table>
-@row{<b>WiredTiger 2.4.1</b> (current),
+@row{<b>WiredTiger 2.5.0</b> (current),
+	<a href="releases/wiredtiger-2.5.0.tar.bz2"><b>[Release package]</b></a>,
+	<a href="2.5.0/index.html"><b>[Documentation]</b></a>}
+@row{<b>WiredTiger 2.4.1</b> (previous),
 	<a href="releases/wiredtiger-2.4.1.tar.bz2"><b>[Release package]</b></a>,
 	<a href="2.4.1/index.html"><b>[Documentation]</b></a>}
-@row{<b>WiredTiger 2.3.1</b> (previous),
-	<a href="releases/wiredtiger-2.3.1.tar.bz2"><b>[Release package]</b></a>,
-	<a href="2.3.1/index.html"><b>[Documentation]</b></a>}
 @row{<b>Development branch</b>,
 	<a href="https://github.com/wiredtiger/wiredtiger"><b>[Source code]</b></a>,
 	<a href="develop/index.html"><b>[Documentation]</b></a>}

src/docs/upgrading.dox

@@ -1,6 +1,6 @@
 /*! @page upgrading Upgrading WiredTiger applications
 
-@section version_242 Upgrading to Version 2.4.2
+@section version_250 Upgrading to Version 2.5.0
 
 <dl>
 <dt>WT_STAT_CONN_LOG_BYTES_USER	renamed WT_STAT_CONN_LOG_BYTES_PAYLOAD</dt>

src/include/extern.h

@@ -310,7 +310,7 @@ extern int __wt_log_open(WT_SESSION_IMPL *session);
 extern int __wt_log_close(WT_SESSION_IMPL *session);
 extern int __wt_log_newfile(WT_SESSION_IMPL *session, int conn_create, int *created);
 extern int __wt_log_read(WT_SESSION_IMPL *session, WT_ITEM *record, WT_LSN *lsnp, uint32_t flags);
-extern int __wt_log_scan(WT_SESSION_IMPL *session, WT_LSN *lsnp, uint32_t flags, int (*func)(WT_SESSION_IMPL *session, WT_ITEM *record, WT_LSN *lsnp, void *cookie), void *cookie);
+extern int __wt_log_scan(WT_SESSION_IMPL *session, WT_LSN *lsnp, uint32_t flags, int (*func)(WT_SESSION_IMPL *session, WT_ITEM *record, WT_LSN *lsnp, void *cookie, int firstrecord), void *cookie);
 extern int __wt_log_write(WT_SESSION_IMPL *session, WT_ITEM *record, WT_LSN *lsnp, uint32_t flags);
 extern int __wt_log_vprintf(WT_SESSION_IMPL *session, const char *fmt, va_list ap);
 extern int __wt_logrec_alloc(WT_SESSION_IMPL *session, size_t size, WT_ITEM **logrecp);

src/log/log.c

@@ -1160,7 +1160,7 @@ err:
 int
 __wt_log_scan(WT_SESSION_IMPL *session, WT_LSN *lsnp, uint32_t flags,
     int (*func)(WT_SESSION_IMPL *session,
-    WT_ITEM *record, WT_LSN *lsnp, void *cookie), void *cookie)
+    WT_ITEM *record, WT_LSN *lsnp, void *cookie, int firstrecord), void *cookie)
 {
 	WT_CONNECTION_IMPL *conn;
 	WT_DECL_ITEM(uncitem);
@@ -1174,6 +1174,7 @@ __wt_log_scan(WT_SESSION_IMPL *session, WT_LSN *lsnp, uint32_t flags,
 	uint32_t allocsize, cksum, firstlog, lastlog, lognum, rdup_len, reclen;
 	u_int i, logcount;
 	int eol;
+	int firstrecord;
 	char **logfiles;
 
 	conn = S2C(session);
@@ -1181,6 +1182,7 @@ __wt_log_scan(WT_SESSION_IMPL *session, WT_LSN *lsnp, uint32_t flags,
 	log_fh = NULL;
 	logcount = 0;
 	logfiles = NULL;
+	firstrecord = 1;
 	eol = 0;
 	WT_CLEAR(buf);
 
@@ -1364,10 +1366,13 @@ advance:
 				WT_ERR(__log_decompress(session, &buf,
 				    &uncitem));
 				WT_ERR((*func)(session, uncitem, &rd_lsn,
-				    cookie));
+				    cookie, firstrecord));
 				__wt_scr_free(&uncitem);
 			} else
-				WT_ERR((*func)(session, &buf, &rd_lsn, cookie));
+				WT_ERR((*func)(session, &buf, &rd_lsn, cookie,
+				    firstrecord));
+
+			firstrecord = 0;
 
 			if (LF_ISSET(WT_LOGSCAN_ONE))
 				break;

src/log/log_auto.c

@@ -135,7 +135,7 @@ __wt_logop_col_put_print(
 	fprintf(out, "    \"fileid\": \"%" PRIu32 "\",\n", fileid);
 	fprintf(out, "    \"recno\": \"%" PRIu64 "\",\n", recno);
 	WT_RET(__logrec_jsonify_str(session, &escaped, &value));
-	fprintf(out, "    \"value\": \"%s\",\n", escaped);
+	fprintf(out, "    \"value\": \"%s\"", escaped);
 	__wt_free(session, escaped);
 	return (0);
 }
@@ -192,7 +192,7 @@ __wt_logop_col_remove_print(
 
 	fprintf(out, "    \"optype\": \"col_remove\",\n");
 	fprintf(out, "    \"fileid\": \"%" PRIu32 "\",\n", fileid);
-	fprintf(out, "    \"recno\": \"%" PRIu64 "\",\n", recno);
+	fprintf(out, "    \"recno\": \"%" PRIu64 "\"", recno);
 	return (0);
 }
 
@@ -250,7 +250,7 @@ __wt_logop_col_truncate_print(
 	fprintf(out, "    \"optype\": \"col_truncate\",\n");
 	fprintf(out, "    \"fileid\": \"%" PRIu32 "\",\n", fileid);
 	fprintf(out, "    \"start\": \"%" PRIu64 "\",\n", start);
-	fprintf(out, "    \"stop\": \"%" PRIu64 "\",\n", stop);
+	fprintf(out, "    \"stop\": \"%" PRIu64 "\"", stop);
 	return (0);
 }
 
@@ -312,7 +312,7 @@ __wt_logop_row_put_print(
 	WT_RET(__logrec_jsonify_str(session, &escaped, &key));
 	fprintf(out, "    \"key\": \"%s\",\n", escaped);
 	WT_RET(__logrec_jsonify_str(session, &escaped, &value));
-	fprintf(out, "    \"value\": \"%s\",\n", escaped);
+	fprintf(out, "    \"value\": \"%s\"", escaped);
 	__wt_free(session, escaped);
 	return (0);
 }
@@ -372,7 +372,7 @@ __wt_logop_row_remove_print(
 	fprintf(out, "    \"optype\": \"row_remove\",\n");
 	fprintf(out, "    \"fileid\": \"%" PRIu32 "\",\n", fileid);
 	WT_RET(__logrec_jsonify_str(session, &escaped, &key));
-	fprintf(out, "    \"key\": \"%s\",\n", escaped);
+	fprintf(out, "    \"key\": \"%s\"", escaped);
 	__wt_free(session, escaped);
 	return (0);
 }
@@ -437,7 +437,7 @@ __wt_logop_row_truncate_print(
 	fprintf(out, "    \"start\": \"%s\",\n", escaped);
 	WT_RET(__logrec_jsonify_str(session, &escaped, &stop));
 	fprintf(out, "    \"stop\": \"%s\",\n", escaped);
-	fprintf(out, "    \"mode\": \"%" PRIu32 "\",\n", mode);
+	fprintf(out, "    \"mode\": \"%" PRIu32 "\"", mode);
 	__wt_free(session, escaped);
 	return (0);
 }

src/txn/txn_log.c

@@ -66,9 +66,22 @@ static int
 __txn_commit_printlog(
     WT_SESSION_IMPL *session, const uint8_t **pp, const uint8_t *end, FILE *out)
 {
+	int firstrecord;
+
+	firstrecord = 1;
+
 	/* The logging subsystem zero-pads records. */
-	while (*pp < end && **pp)
+	while (*pp < end && **pp) {
+		if (!firstrecord)
+			fprintf(out, ",\n");
+
+		firstrecord = 0;
+
 		WT_RET(__wt_txn_op_printlog(session, pp, end, out));
+	}
+
+	fprintf(out, "\n");
+
 	return (0);
 }
 
@@ -412,8 +425,8 @@ __wt_txn_truncate_end(WT_SESSION_IMPL *session)
  *	Print a log record in a human-readable format.
  */
 static int
-__txn_printlog(
-    WT_SESSION_IMPL *session, WT_ITEM *rawrec, WT_LSN *lsnp, void *cookie)
+__txn_printlog(WT_SESSION_IMPL *session,
+    WT_ITEM *rawrec, WT_LSN *lsnp, void *cookie, int firstrecord)
 {
 	FILE *out;
 	WT_LOG_RECORD *logrec;
@@ -435,6 +448,9 @@ __txn_printlog(
 	/* First, peek at the log record type. */
 	WT_RET(__wt_logrec_read(session, &p, end, &rectype));
 
+	if (!firstrecord)
+		fprintf(out, ",\n");
+
 	if (fprintf(out, "  { \"lsn\" : [%" PRIu32 ",%" PRId64 "],\n",
 	    lsnp->file, lsnp->offset) < 0 ||
 	    fprintf(out, "    \"hdr_flags\" : \"%s\",\n",
@@ -448,16 +464,16 @@ __txn_printlog(
 	case WT_LOGREC_CHECKPOINT:
 		WT_RET(__wt_struct_unpack(session, p, WT_PTRDIFF(end, p),
 		    WT_UNCHECKED_STRING(IQ), &ckpt_lsn.file, &ckpt_lsn.offset));
-		if (fprintf(out, "    \"type\" : \"checkpoint\"\n") < 0 ||
+		if (fprintf(out, "    \"type\" : \"checkpoint\",\n") < 0 ||
 		    fprintf(
-		    out, "    \"ckpt_lsn\" : [%" PRIu32 ",%" PRId64 "],\n",
+		    out, "    \"ckpt_lsn\" : [%" PRIu32 ",%" PRId64 "]\n",
 		    ckpt_lsn.file, ckpt_lsn.offset) < 0)
 			return (errno);
 		break;
 
 	case WT_LOGREC_COMMIT:
 		WT_RET(__wt_vunpack_uint(&p, WT_PTRDIFF(end, p), &txnid));
-		if (fprintf(out, "    \"type\" : \"commit\"\n") < 0 ||
+		if (fprintf(out, "    \"type\" : \"commit\",\n") < 0 ||
 		    fprintf(out, "    \"txnid\" : %" PRIu64 ",\n", txnid) < 0)
 			return (errno);
 		WT_RET(__txn_commit_printlog(session, &p, end, out));
@@ -466,8 +482,8 @@ __txn_printlog(
 	case WT_LOGREC_FILE_SYNC:
 		WT_RET(__wt_struct_unpack(session, p, WT_PTRDIFF(end, p),
 		    WT_UNCHECKED_STRING(Ii), &fileid, &start));
-		if (fprintf(out, "    \"type\" : \"file_sync\"\n") < 0 ||
-		    fprintf(out, "    \"fileid\" : %" PRIu32 "\n",
+		if (fprintf(out, "    \"type\" : \"file_sync\",\n") < 0 ||
+		    fprintf(out, "    \"fileid\" : %" PRIu32 ",\n",
 		    fileid) < 0 ||
 		    fprintf(out, "    \"start\" : %" PRId32 "\n", start) < 0)
 			return (errno);
@@ -476,13 +492,13 @@ __txn_printlog(
 	case WT_LOGREC_MESSAGE:
 		WT_RET(__wt_struct_unpack(session, p, WT_PTRDIFF(end, p),
 		    WT_UNCHECKED_STRING(S), &msg));
-		if (fprintf(out, "    \"type\" : \"message\"\n") < 0 ||
+		if (fprintf(out, "    \"type\" : \"message\",\n") < 0 ||
 		    fprintf(out, "    \"message\" : \"%s\"\n", msg) < 0)
 			return (errno);
 		break;
 	}
 
-	if (fprintf(out, "  },\n") < 0)
+	if (fprintf(out, "  }") < 0)
 		return (errno);
 
 	return (0);
@@ -503,7 +519,7 @@ __wt_txn_printlog(WT_SESSION *wt_session, FILE *out)
 		return (errno);
 	WT_RET(__wt_log_scan(
 	    session, NULL, WT_LOGSCAN_FIRST, __txn_printlog, out));
-	if (fprintf(out, "]\n") < 0)
+	if (fprintf(out, "\n]\n") < 0)
 		return (errno);
 
 	return (0);

src/txn/txn_recover.c

@@ -265,8 +265,8 @@ __txn_commit_apply(
  *	Roll the log forward to recover committed changes.
  */
 static int
-__txn_log_recover(
-    WT_SESSION_IMPL *session, WT_ITEM *logrec, WT_LSN *lsnp, void *cookie)
+__txn_log_recover(WT_SESSION_IMPL *session,
+    WT_ITEM *logrec, WT_LSN *lsnp, void *cookie, int firstrecord)
 {
 	WT_RECOVERY *r;
 	const uint8_t *end, *p;
@@ -276,6 +276,7 @@ __txn_log_recover(
 	r = cookie;
 	p = LOG_SKIP_HEADER(logrec->data);
 	end = (const uint8_t *)logrec->data + logrec->size;
+	WT_UNUSED(firstrecord);
 
 	/* First, peek at the log record type. */
 	WT_RET(__wt_logrec_read(session, &p, end, &rectype));

test/suite/test_config02.py

@@ -149,6 +149,8 @@ class test_config02(wttest.WiredTigerTestCase):
             '/No such file or directory/')
 
     def test_home_not_writeable(self):
+        if os.name == "nt":
+            self.skipTest('Unix specific test skipped on Windows')
         dir = 'subdir'
         os.mkdir(dir)
         os.chmod(dir, 0555)