Commit Graph

18269 Commits

Author SHA1 Message Date
Michael Cahill
fb1663e6fc Merge branch 'develop' into mongodb-3.4 mongodb-3.3.8 mongodb-3.3.7 2016-06-03 15:18:38 +10:00
Michael Cahill
c9dfa31598 WT-2325 Fix an incomplete comment. 2016-06-03 14:23:11 +10:00
Michael Cahill
7c4a0a1f59 WT-2629 Force non-executable stacks based on __ELF__. (#2774) 2016-06-03 13:53:32 +10:00
sueloverso
c256c8302b SERVER-23659 Improve log error message at startup. (#2766) 2016-06-03 13:34:45 +10:00
Keith Bostic
d3a8060e5b WT-2658 Only include PPC-specific files in PPC builds (#2758)
* WT-2658 Only include PPC-specific files in PPC builds

* Add support for conditional file inclusion by adding a second argument on lines in dist/filelist.  Add a new automake conditional, POWERPC_HOST, set for the various PPC host CPUs.  Change dist/filelist to only compile the checksum/power8 files if POWERPC_HOST is set.

* Merge the Windows and POSIX file lists, use POSIX_HOST and WINDOWS_HOST to mark source files needed for builds on those systems.
2016-06-03 12:54:40 +10:00
sueloverso
a27db763dc WT-2589 Clear the cursor statistics for LAS when 'clear' is set. (#2772) 2016-06-02 16:30:39 -04:00
Keith Bostic
ced588aecd WT-2672 handle system calls that don't set errno (#2765)
Define system call success as a 0 return, and split error handling into two parts: if the call returns -1, use errno, otherwise expect the failing return to be an error value.

Replace calls to remove with unlink, so we know errno will be set.  Do the best we can with rename, there's no easy workaround.

POSIX requires posix_madvise return an errno value, but some OS versions return a -1/errno pair instead (at least FreeBSD and OS X). I don't care about retrying posix_madvise calls on failure, but since WT_SYSCALL_RETRY includes the necessary error handling magic, wrap the posix_madvise calls in WT_SYSCALL_RETRY.
2016-06-02 10:02:33 +10:00
Keith Bostic
16ecfdbfdb Revert "WT-2677 Fix JSON output so only printable ASCII is produced. (#2764)"
This reverts commit 4d19d24ce3.
2016-06-01 17:24:18 -04:00
Don Anderson
4d19d24ce3 WT-2677 Fix JSON output so only printable ASCII is produced. (#2764)
The root cause is a sign extension error that shows up on systems
that have a signed 'char'.
2016-06-01 16:37:35 -04:00
Alex Gorrod
ff108d7c70 Merge branch 'develop' into mongodb-3.4 2016-06-01 12:38:19 +10:00
Keith Bostic
7033ed47f3 WT-2674 simplify metadata file check (#2763)
Remove the need for a hash and name comparison to identify the metadata file, set a flag in the data handle when the file is opened.

Move the code to insert the data handle into the connection list from the find function to the allocate function for clarity.
2016-06-01 10:30:18 +10:00
Michael Cahill
b89aaece7b SERVER-24306 Fix stall in log_flush switching to new files. (#2761)
* SERVER-24306 Fix stall in log_flush switching to new files.

* Pass boolean false rather than 0.
2016-05-31 09:31:47 -04:00
Keith Bostic
564f8703ad WT-2673 Stop automatically increasing memory page max (#2762)
btree->maxmempage = cval.val;
src/btree/bt_handle.c  698  Info 732: Loss of sign (assignment)
(long long to unsigned long long)
2016-05-31 08:02:27 -04:00
Alex Gorrod
5a11406ae9 WT-2673 Stop automatically increasing memory page max (#2760)
We used to ensure it was at least 50 times the leaf page size, enhancements to how we split pages mean that's no longer necessary.

While reviewing the code adjust the maximum page size allowed to ten percent of the cache size. It is more reasonable for workloads with multiple tables, and should only be relevant when applications configure tiny caches.
2016-05-31 03:19:04 -04:00
Don Anderson
27981762ba WT-2268 WT-2597 JSON load/dump Unicode fixes (#2749)
* WT-2268 Change dump representation so every byte represents a single 0x00-0xff
Unicode character. Otherwise arbitrary binary data (that does not conform to
Unicode) cannot be represented. Fix an off-by-one error in counting bytes
on input. Added some better Unicode tests.

* JSON Dump now uses a 'dump version' stamp, as well as showing WT version.
Dump input that is too old, and input that is too new will be rejected.

* In setting a key for dump, converting a JSON string must always occur first.

* When loading JSON, treat binary data just like strings.
Fixed an error in determining the string length of JSON in the presence
of Unicode strings.

* Add LSM tests to JSON dump/load testing.

* Add more extensive testing for dump/reload of JSON binary data.

* Byte arrays differ from strings for JSON input, they do not null terminate.
Handle escapes like '\n', '\t', etc. on input.

* Added tests of JSON dump/load of all byte codes.

* whitespace
2016-05-31 11:39:51 +10:00
David Hows
12b772612a WT-2491 Revert - Remove the handle close_lock (#2759)
This reverts commit fa4b0916bc.
2016-05-30 21:36:08 -04:00
David Hows
fa4b0916bc WT-2491 Remove the handle close_lock (#2745)
It is no longer necessary, due to other structural code changes.
2016-05-30 03:42:06 -04:00
Alex Gorrod
6f9a7a41f2 Merge branch 'develop' into mongodb-3.4 2016-05-27 15:50:49 +10:00
sueloverso
4d13c2b911 WT-2660 close hang (#2752)
* WT-2660 Fix eviction hang when closing.

* Real fix: return if we're closing the connection.

* Move check for connection close into wt_eviction_needed.

* Fill in pct_full before returning.

* Whitespace

* WT-2660 Uninitialized variable
2016-05-26 15:44:32 -04:00
sueloverso
5fe33c475e WT-2660 Fill in pct_full before returning. (#2751)
* WT-2660 Fix eviction hang when closing.

* Real fix: return if we're closing the connection.

* Move check for connection close into wt_eviction_needed.

* Fill in pct_full before returning.

* Whitespace
2016-05-26 14:26:57 -04:00
sueloverso
eb9cf0388b WT-2660 Fix a hang between eviction and connection close (#2747)
* Fix eviction hang when closing.

* Real fix: return if we're closing the connection.

* Move check for connection close into wt_eviction_needed.
2016-05-26 11:19:48 -04:00
Don Anderson
230fed7fe8 WT-2662 For spell check, strip out double quote literals, they confuse aspell. (#2748) 2016-05-25 15:09:36 -04:00
Keith Bostic
5c70c81687 Wt 2661 coverity (#2746)
* WT-2661: Coverity failures: 1356050-1356053

CID 1356050: (NULL_RETURNS)
Dereferencing a pointer that might be null "opts->home" when calling "snprintf".
CID 1356050: (NULL_RETURNS)
Dereferencing a pointer that might be null "opts->uri" when calling "snprintf".

* Assigning address of auto variable '_opts' to static (it's an auto
variable in main(), so it's safe, but make lint happy).

* Coverity appears to be confused by argv handling and eventually passing
NULL to snprintf.
	2. var_assign_alias: Assigning: opts->progname = argv[0]. Both
	are now tainted.

	47. vararg_transitive: Call to snprintf with tainted argument
	opts->progname taints NULL.

	48. tainted_data_sink_lv_call: Passing tainted variable 0UL to
	tainted data sink snprintf.

NULL is tainted? I'm hoping that using a strlen() call to figure out the
string length instead of snprintf() will make this one go away.

* test_wt2246_col_append wasn't actually doing any work, set nrecords to
a 20 million default.

* KNF

* Don't use "inline" in example programs, it requires Windows #ifdef's.
2016-05-25 12:25:13 -04:00
Keith Bostic
c0a04f1d22 WT-2653: the custom file-system example should show device configuration (#2741)
Add examples of passing in per-device configuration information to the demo file system initialization function.

Use the WiredTiger extension API to handle errors in functions called by WiredTiger.

Set the default session's strerror method. If one of the extensions being loaded reports an error via the WT_EXTENSION_API strerror method, but doesn't supply that method a WT_SESSION handle, we'll use the WT_CONNECTION_IMPL's default session and its strerror method.
2016-05-25 09:40:00 +10:00
Keith Bostic
ea8840e93c WT-2659: csuite tests, assorted lint and cleanup. (#2744)
* WT-2659: csuite tests, assorted lint and cleanup.

Fail if memory allocation fails, ignore usleep() and pthread_join()
returns.

* Don't check for NULL before calling free, free does that for you.

* Convert (int) casts to the correct printf format string, KNF.

* Assert that ret == WT_NOTFOUND when exiting the WT_CURSOR.next loop
(Warning 438: Last value assigned to variable 'ret' (defined at line
122) not used).

Add testutil_assert macro to make it easy to assert boolean values.

* Explicit global initialization.

* Use correct printf format types instead of casting to (int), KNF.

* Remove unused variable.

* Fix testutil_assert() to lint cleanly.

* Assert various WiredTiger functions succeed.
2016-05-24 10:39:38 -04:00
David Hows
6f3fdd2444 WT-2656 Fix GCC 4.7 compiler warnings (#2743) 2016-05-24 17:19:24 +10:00
Michael Cahill
f6f86961a4 WT-2629 Make the stack non-executable with GCC only. (#2742) 2016-05-24 16:24:48 +10:00
David Hows
4b48ad6fb7 WT-2646 Add checkpoint_wait configuration option to drop (#2736)
* Default checkpoint_wait is true. This change is useful because it means concurrent create/drop calls don't generate EBUSY returns.
* Mark lock_wait and checkpoint_wait as undoc
2016-05-24 14:45:53 +10:00
Michael Cahill
cace8acbbe WT-2630 Rename pluggable filesystem methods to avoid reserved names. (#2730)
* WT-2630 Rename pluggable filesystem methods to avoid reserved names.

Use "fstr" consistently as the variable name and prefix for WT_FSTREAM.
2016-05-24 12:20:39 +10:00
sueloverso
2c16e4a8a6 WT-2554 Fix build warnings (#2740) 2016-05-23 16:27:25 -04:00
Alex Gorrod
9768c2db0b WT-2554 Add initial C test framework. (#2655)
* WT-2554 Add initial C test framework.

With a few sample applications. Need to do further work to cleanup
the same applications and share as much code/style as possible.

* WT-2554 Fixup build errors.

* Fix compiler warning.

* Cleanup

* Ensure each test runs in a different directory.

* Update C test suite names and add copyright.

* Move test code into subdirectories.

* Replace test_util.i with a real utility library now that we have one.

Nothing needs to be an inline function.

Mostly cherry picked from 7c18420.

* Use WiredTiger getopt in C test suite for platform portability.

* Add a header comment to each test case.

* Add a C test suite entry auto generator

* Style, KNF

* Implement review feedback.

* Build test/utility library on Windows

* Add comment to script.  Fix a printf.
2016-05-23 13:13:54 -04:00
Michael Cahill
0f7ae730d9 WT-2629 Don't make stacks executable in assembly source. (#2739) 2016-05-23 06:42:49 -04:00
Keith Bostic
eeff19c80c WT-2651: Coverity 1355591 resource leak (#2738)
Fix comment spelling.
2016-05-22 09:48:25 -04:00
Keith Bostic
1b9a685898 WT-2651: Coverity 1355591 resource leak (#2737)
* Free memory when returning an error.
* Remove unnecessary parenthesis.
* Make repeated addition/subtraction operator order of evaluation explicit.
* Remove repeated #includes.
* Some minor restructuring of the config file parsing to handle more
complicated line continuation and whitespace combinations.
2016-05-22 08:13:49 -04:00
Don Anderson
40b21968b8 WT-2611 Modify wtperf to handle escaped quotes. (#2714)
* Modify wtperf to handle escaped quotes.

* Made local function static.
Added backslash in quote (normally redundant) needed to keep spell check sane.
Fixed C++ style comment.
2016-05-19 16:53:57 -04:00
Don Anderson
df54fd30bd WT-2644 Fix 'wt load -r' (rename) with LSM (#2735)
* WT-2644 recognize lsm: URLs in the list of items to be renamed.

* Add a test to load a dump file, renaming the url. Fix other dump/load tests,
as the populate_check methods always use self.session. Up to now they were only
confirming that tables in the current directory were correct, not tables
in the dump.dir directory.
2016-05-19 14:48:16 -04:00
sueloverso
9a6329b560 WT-2632 Tolerate EBUSY when a checkpoint cursor is open. (#2734)
* WT-2632 Tolerate EBUSY when a checkpoint cursor is open.

* Tolerate EBUSY in general.
2016-05-19 14:44:15 -04:00
Keith Bostic
88801726cc WT-2637: The file-extension documentation doesn't cover not-supported cases (#2733)
* WT-2637: The file-extension documentation doesn't cover not-supported cases

Clarify not-supported cases in the file-extension documentation.

Public domain the POSIX file system implementation, it's a natural
starting point for anyone writing their own implementation.

* Fix a typo, add some wording around when methods can be changed after
hnadle creation.

* Add a pointer to ex_file_system.c, the principle custom file system
example.
2016-05-19 12:13:11 -04:00
sueloverso
850bfde293 WT-2610 Reduce hazard pointer array size. (#2731)
* WT-2610 Reduce hazard pointer array size.

* Add session_count_idle and workload to wtperf.

* Allocate session array outside the loop.

* KNF

* Remove diagnostic call to check hazard pointers.

* Add a few hazard related statistics.

* Revert change.  Reinstate diagnostic hazard page check.
2016-05-17 17:08:51 -04:00
Keith Bostic
0893547e78 WT-2628 Reconciliation can return without unlocking the page lock (#2728)
Don't clean up reconciliation structures before they've been initialized.

Fix two paths where scratch buffers could be allocated and fail to be
released.
2016-05-16 11:13:05 -04:00
Keith Bostic
62fc6016ee WT-2627: Coverity complaints (#2727)
* WT-2627: Coverity complaints

Coverity complaint 1354814: error path leaking allocated memory.

Initially clear caller's handle in __wt_fopen().

* Coverity 1355027 thinks there's a path to leak memory in:

	__backup_list_uri_append()
	calls __wt_metadata_search()
	calls__wt_turtle_read()
	calls __metadata_config()
	calls __wt_config_collapse()

which I think should only happen if one of those functions allocates
memory and then returns an error, and I don't see it.

Change __wt_config_collapse to always clear the caller's
memory location, it's good practice.

Simplify __metadata_config slightly, we don't need need a local variable
into which we store the allocated memory, and there's no error path that
might need to free that allocated memory, hope that helps Coverity.

* Whitespace, KNF.

* Whitespace
2016-05-09 11:33:19 -04:00
David Hows
7153550cce WT-2626 Fix MSVC 2015 snprintf redefinition (#2726) 2016-05-06 17:01:50 +10:00
David Hows
b1cebb044a WT-2624 Fix example program build with MSVC 2013 (#2725)
Since Visual Studio 2013 doesn't have a snprintf symbol.
2016-05-06 15:31:40 +10:00
Alex Gorrod
8ac9a3057b WT-2617 Don't use u_int in the example code. It's not Windows native. (#2722) 2016-05-05 17:22:52 +10:00
Keith Bostic
8c183b0772 WT-2617 Fix a bug in pluggable file system example (#2718)
Don't print "home" if it's NULL.

Make ex_async work like the rest of the example programs, creating and
using WT_HOME.

* We need two sizes, the size of the data in the buffer and the size of the
buffer itself, split the two apart, otherwise it's possible to read data
that's never been written.

Remove the assert the file size is never 0, that's a reasonable condition
if the file has never been written.

Don't bother realloc the buffer when truncating it to a smaller size.

* Add a full smoke test, create a table then write it, close and reopen
it, and read the values back.

* main() should return EXIT_SUCCESS or EXIT_FAILURE, not a return value
from the underlying WiredTiger library.
2016-05-05 16:14:34 +10:00
Michael Cahill
f117f2c271 Merge branch 'master' into develop 2016-05-05 15:39:41 +10:00
Michael Cahill
636a7b25ef Merge branch 'develop' into mongodb-3.4 mongodb-3.3.6 2016-05-05 15:38:12 +10:00
David Hows
75c22bc0c6 WT-2621 Fix missing parentesies on call to __wt_errno (#2721) 2016-05-05 15:17:05 +10:00
Keith Bostic
55505b5cbc WT-2616 Fix a deadlock with in-memory size lookups. (#2719)
Eliminate a race where the handle could be removed between the search
and the size check.
2016-05-05 09:32:21 +10:00
Keith Bostic
2437ec597d WT-2615 Enabling checkpoints in test/format leads to reduced concurrency (#2720)
Don't queue threads for checkpoint operations, just skip the scheduled
checkpoint operation if a checkpoint is already in progress.

Don't wait for backups to complete if scheduled to do a named checkpoint,
do an unnamed checkpoint instead.

Fix a bug: we were scheduling named checkpoints 80% of the time instead
of the documented 20%; change that number to 5%, named checkpoints aren't
worth testing that heavily.

Don't bother avoiding scheduling two named checkpoints in a row, it's
not worth the effort.
2016-05-05 09:25:01 +10:00