* Default checkpoint_wait is true. This change is useful because it means concurrent create/drop calls don't generate EBUSY returns.
* Mark lock_wait and checkpoint_wait as undoc
* WT-2552 Add public API for pluggable filesystems
Not yet compiling. The main parts of this change should be here,
but it involved extensive parameter re-organization. There are also
a number of layering violations between our existing file system
implementations and the WT_FH, that aren't possible with the new
structure.
There are a number of specific todo comments in the code. One of the main
issues is that the in-memory file system had a special close semantic
that relied on WiredTiger handle tracking. The in-memory file-system should
do it's own tracking of file handles, I've gone part way down that road by
adding a queue for closed handles. Need to also add in live handles, and
manage the queue as appropriate.
I haven't created an example application that uses the new API yet.
* WT-2552 Add public API for pluggable filesystems
I always forget you have to remove the already-built html files when
changing PREDEFINED, add a reminder to the complaint.
* WT-2552 Add public API for pluggable filesystems
You have to remove the .js files, too.
* WT-2552 Add public API for pluggable filesystems
Make dist/s_all run cleanly.
* WT-2552 Add public API for pluggable filesystems
Whitespace.
* WT-2552 Add public API for pluggable filesystems
Make it compile/build/lint.
* WT-2552 Add public API for pluggable filesystems
block_write.c: In function '__wt_block_extend':
block_write.c:130:71: error: missing terminating ' character [-Werror]
* WT-2552 Add public API for pluggable filesystems
os_fs_inmemory.c: In function '__im_file_truncate':
os_fs_inmemory.c:344:10: error: 'session' is used uninitialized in this
function [-Werror=uninitialized]
* WT-2552 Add public API for pluggable filesystems
os_fs.c: In function '__posix_directory_sync':
os_fs.c:92:10: error: 'session' is used uninitialized in this function
[-Werror=uninitialized]
* WT-2552 Add public API for pluggable filesystems
Go back to using bool types in the file-system API, this requires we add
<stdbool.h> to the "standard" wiredtiger.h includes.
Consistently use wt_session to represent a WT_SESSION, we were using
"wtsession" in some places.
Make a pass over the Windows code, but I'm sure it doesn't compile yet.
* WT-2552 Add public API for pluggable filesystems
Fix up another couple of bool types.
* WT-2552 Add public API for pluggable filesystems
Move the file naming work out of the underlying filesystem functions,
the calls to __wt_filename are now in the upper-level code,n os_fs.i;
that means the filesystem code is no longer responsible for figuring out
paths. This is cleaner, although the directory-sync call is a bit of a
kluge, and I've commimtted us to handling NULL filesystem methods.
With this set of changes, in-memory runs again.
More Windows naming fixes.
* WT-2552 Add public API for pluggable filesystems
os_fs.c: In function '__posix_directory_sync':
os_fs.c:96:3: error: label 'err' used but not defined
* WT-2552 Add public API for pluggable filesystems
Pull out another call to __wt_filename() from the filesystem-dependent
code.
* WT-2552 Add public API for pluggable filesystems
Consistently check for missing file-system methods when doing
file-system calls.
Other minor lint & cleanup.
* WT-2552 Add public API for pluggable filesystems
Change the in-memory code to maintain a complete list of the files it
has ever opened, and depend on that list instead of reaching up into the
common layer for the WT_FH handle list.
This means __wt_handle_search is only used by the common WT_FH handle
code, simplify it, and add a __wt_handle_is_open function that can be
called for diagnostic purposes (to check for open files that are being
renamed or removed, for example).
* Fix comiler warning and ignore the file system API in Java
* Flesh out the example file system implementation.
* Add in some plumbing for set_file_system in wiredtiger_open.
* WT-2552 Add public API for pluggable filesystems
Whitespace.
* WT-2552 Add public API for pluggable filesystems
WT_CONFIG_ITEM.val isn't a boolean, don't use boolean types in
equal/not-equal comparisons.
* WT-2552 Add public API for pluggable filesystems
Remove unused #includes.
Increment/decrement the DEMO_FILE_SYSTEM.{opened,closed}_file_count.
Allocate demo structures, they're larger than the underlying structures.
Swap the number/size calloc arguments, number comes first.
Fix a couple of statics.
* WT-2552 Add public API for pluggable filesystems
Use %u instead of casting to %d.
* WT-2552 Add public API for pluggable filesystems
Add ex_file_system.c to the list of example programs.
* WT-2552 Add public API for pluggable filesystems
Change ex_file_system.c to not require <wt_internal.h>: strip down a
copy of FreeBSD's <queue.h> for local inclusion, rewrite a few other
minor pieces of code.
* WT-2552 Add public API for pluggable filesystems
Update spell check info
* WT-2552 Add public API for pluggable filesystems
__conn_load_extensions() shouldn't set the "early" boolean to true.
* WT-2552 Add public API for pluggable filesystems
Don't indirect through a NULL pointer if "local" was set and no path was
specified, always set the name to something useful.
* WT-2552 Add public API for pluggable filesystems
Don't indirect through a NULL pointer if "local" was set and no path was
specified, always set the name to something useful.
* WT-2552 Add public API for pluggable filesystems
wt_off_t vs. size_t conversion lint.
* WT-2552 Add public API for pluggable filesystems
Add -rdynamic to the load for ex_file_system, the main executable
symbols are not exported by default.
* WT-2552 Add public API for pluggable filesystems
The underlying handle name includes the enclosing directory,
compare against the WT_FH.name field instead.
* WT-2552 Add public API for pluggable filesystems
demo_fs_rename should return 0 if successful, simplify error handling
Don't bother casting arguments to free(), it's not necessary.
* WT-2552 Add public API for pluggable filesystems
General WT_FILE_SYSTEM cleanup.
Move OS initialization into the wiredtiger_open() code (the
os_common/os_init.c file is no longer needed).
Allow early-load extensions to be part of the environment settings,
matching the "in-memory" and "readonly" configurations.
Syntax check the set of a file-system, remove tests for NULL methods in
the file-system structure unless it's legal for them to be NULL.
Windows, POSIX and in-memory file systems now set WT_FILE_SYSTEM.terminate,
call that function to cleanup when discarding a WT_CONNECTION.
Export file-type and open-flags constants for WT_FILE_SYSTEM.open_file,
sort the WT_FILE_SYSTEM methods, do an editing pass.
Change the WT_FILE_HANDLE type from (const char *) to (char *), it's
"owned" by the underlying layer, and it's simpler that way.
Minor (untested) cleanup of the Windows WT_FILE_SYSTEM.open-file method.
* WT-2552 Add public API for pluggable filesystems
Export the advise argument #defines for the WT_FILE_HANDLE.fadvise method.
Sort the WT_FILE_HANDLE methods.
* WT-2552 Add public API for pluggable filesystems
Clean up and simplify WT_FILE_SYSTEM/WT_FILE_HANDLE documentation's
description of the handles.
* WT-2552 Add public API for pluggable filesystems
WT_FILE_HANDLE.close is a required function (at the least, it
has to free the memory).
WT_FILE_HANDLE.fadvise isn't a required function, if it's not
configured, don't call it.
* WT-2552 Add public API for pluggable filesystems
The WT_FILE_HANDLE.lock function is required.
Change the __wt_open() signature to match WT_FILE_SYSTEM.open_file().
* WT-2552 Add public API for pluggable filesystems
Rework all of the WT_FILE_HANDLE mapped region methods to be optional.
* WT-2552 Add public API for pluggable filesystems
The WT_FILE_HANDLE.{read,size} methods are required.
The WT_FILE_HANDLE.sync method is not required.
Split the WT_FILE_HANDLE.sync method into .sync and .sync_nowait versions,
it makes the upper-level code simpler (Windows supports .sync but doesn't
support .sync_nowait).
* WT-2552 Add public API for pluggable filesystems
The WT_FILE_HANDLE.{truncate,write} methods are required IFF the file
is not readonly.
* WT-2552 Add public API for pluggable filesystems
POSIX shouldn't declare a no-sync handle function unless the
sync_file_range system call is available.
* WT-2552 Add public API for pluggable filesystems
Typo, missing semi-colon.
* Fix a bug in ex_file_system.c
* Fix a memory leak in posix file handle implementation
* WT-2552 Use the correct flags when opening backup file.
* WT-2552 Add public API for pluggable filesystems
Simplify open-file error handling by calling the close function on the
handle, that way we won't forget to free all of the applicable memory
allocations.
* WT-2552 Add public API for pluggable filesystems
Simplify the directory-list method, don't pass in an include/exclude
file, if prefix is non-NULL, it implies we only want files matching
the prefix.
* WT-2552 Add public API for pluggable filesystems
Replace WT_FILE_HANDLE_POSIX.fallocate_{available,requires_locking} wiht
WT_FILE_HANDLE.fallocate and WT_FILE_HANDLE.fallocate_nolock.
Example code doesn't need to set WT_FILE_HANDLE methods to NULL, the
allocation does that.
Free the I/O buffer if open-handle allocation fails in the example code.
Remove snippets for WT_FILE_SYSTEM and WT_FILE_HANDLE methods, we're
not going to provide example code for them.
* WT-2552 Add public API for pluggable filesystems
Document we expect either ENOTSUP or EBUSY from optionally supported
APIs. Review/cleanups ENOTSUP/EBUSY returns from optionally supported
APIs.
Make WT_FILE_HANDLE.lock optional.
Don't configure or call the POSIX fadvise function on files configured
for direct I/O.
Rename __wt_filesize_name to __wt_size for consistency.
Update the spelling list.
* WT-2552 Add public API for pluggable filesystems
WT_FILE_HANDLE.truncate requires locking in all known implementations,
document it is not called concurrently with other operations.
* WT-2552 Add public API for pluggable filesystems
Don't terminate the filesystem unless we've actually configured one.
* WT-2552 Add public API for pluggable filesystems
Remove WT_FILE_SYSTEM and WT_FILE_HANDLE from SWIG so the test suite
can pass again.
* WT-2552 Add public API for pluggable filesystems
Merge __conn_load_early_extensions() and __conn_load_extensions().
Fix a problem where I moved the early extensions load to where it could
include the WiredTiger environment variable, but I didn't pass the built
cfg into the function.
* WT-2552 Add public API for pluggable filesystems
Linux build typo.
* WT-2552 Add public API for pluggable filesystems
Get rid of the "bool silent" argument to WT_FILE_SYSTEM.size by testing
for the file's existence before requesting the size (an extra system
call, but guaranteed to hit in the buffer cache at least).
* WT-2552 Add public API for pluggable filesystems
Naming consistency pass over the WT_FILE_SYSTEM functions.
* WT-2552 Add public API for pluggable filesystems
Fix a spin lock mismatch.
* WT-2552 Add public API for pluggable filesystems
Another spinlock mismatch.
* Update example pluggable file system.
Add a directory list implementation to the example, which uncovered
an issue with the API. The directory list API allocates memory that
is freed by WiredTiger, which I don't think is kosher.
* Change file-directory-sync to use reguar fsync.
The distinction in os_fs.i doesn't work with the filesystem API.
Also add directory_sync application to the example application.
* WT-2552 Add public API for pluggable filesystems
Whitespace.
* WT-2552 Add public API for pluggable filesystems
Rewrite __wt_free to not evaluate macro arguments multiple times.
* WT-2552 Add public API for pluggable filesystems
Simplify the directory-list functions: __wt_realloc_def() already
handles scaling the size of the allocations, there's no need to
involve a separate constant that increments the allocation size.
* WT-2552 Add public API for pluggable filesystems
Fix a grouping problem in a realloc call, we need to multiple the size
times the previously allocated slots + 10.
Fix buffer overrun, if "count" has already been incremented, the memset
would skip clearing the first slot and clear one slot past the end of
the buffer.
Remove a comment, realloc requires clearing allocated memory, it's not
paranoia.
* WT-2552 Add public API for pluggable filesystems
Add the mapping-cookie argument to the map-preload and map-discard
functions.
Change page-discard to stop reaching down through the block manager,
instead, provide a block-manager map-discard function that does the
work.
* WT-2552 Add public API for pluggable filesystems
Require a directory-list function.
Implement a directory-list function for the in-memory filesystem.
Consistency pass, make all the directory-list functions look the same.
* WT-2552 Add public API for pluggable filesystems
The WT_FILE_SYSTEM.{directory_sync, remove, rename} methods are not
required for read-only systems.
* WT-2552 Add public API for pluggable filesystems
Change the WT_FILE_SYSTEM.open_file file_type argument from a set of
constants to an enum.
This requires changing how we store connection direct I/O configuration
(the constants used to be flags stored in the WT_CONNECTION_IMPL), and
requiring all callers of __wt_open() do their own work to figure out if
WT_OPEN_DIRECTIO should be specified.
* WT-2552 Add public API for pluggable filesystems
Make no guarantees WT_FILE_SYSTEM and WT_FILE_HANDLE methods are
not called concurrently (except for WT_FILE_HANDLE::fallocate and
WT_FILE_HANDLE::fallocate_nolock).
Rewrite the in-memory FS code to lock across all methods (for example,
WT_FILE_HANDLE.close), that means including a reference to the enclosing
WT_FILE_SYSTEM in the WT_FILE_HANDLE structure so we can find a lock
without using the WT_CONNECTION_IMPL structure.
* WT-2552 Add public API for pluggable filesystems
Remove __wt_directory_sync_fh, it's no longer useful.
* WT-2552 Add public API for pluggable filesystems
Rename WT_INMEMORY_FILE_SYSTEM to WT_FILE_SYSTEM_INMEM, matching
WT_FILE_HANDLE_INMEM.
* WT-2552 Add public API for pluggable filesystems
Add WT_FILE_SYSTEM.directory_list_free, to free memory allocated
by WT_FILE_SYSTEM.direct_list.
Fix a memory leak in __log_archive_once (if __wt_readlock failed,
we leaked the directory-list memory).
* WT-2552 Add public API for pluggable filesystems
Typo, check WT_DIRECT_IO_LOG, not WT_DIRECT_IO_CHECKPOINT.
* WT-2552 Add public API for pluggable filesystems
Typo, unreachable code.
* WT-2552 Add public API for pluggable filesystems
We don't require WT_FILE_SYSTEM.{remove,rename} if the system is
read-only.
* Fix Windows build with pluggable file system.
Involved removing u_int from the public API.
* Fix line wrapping.
* Fix Windows terminate function.
* Forgot something in my last commit.
* Fix Windows munmap bug.
* Add new example to Windows build. Extend example to be more complete.
* Fix example loading on Windows
* Update documentation
* Add missing spell words
* Remove old comment.
The join iterator now manages iteration at 'one level' of the join, as
subjoins are iterated, new iterators are created, forming a stack.
Added internal documentation in cursor.h.
Include all of the WT_SESSION::create config in the ordinary LSM
metadata so it is merged correctly into the dump header. Provide
an upgrade path for LSM metadata in the old format.
** Backwards bracking change for LSM: ** once metadata is upgraded
to the new format, LSM trees cannot be opened with older versions of
WiredTiger.
specifies a percentage of the tree to skip over before retrieving each
new record. If next_random_sample_percent is configured, we first
retrieve a random record from the tree as a whole, and on subsequent
retrievals, skip forward the specified percentage of the tree; if
next_random_sample_percent is not configured, we use the previous method
of choosing a new leaf page from the entire tree on each retrieval.
Clarify bulk and next-random cursor documentation: there are several
methods allowed on bulk and next-random cursors (for example, reset,
reconfigure and close), don't try to list them. Instead reference the
principle method the application is expected to use (insert for bulk
cursors, next for next-random cursors).
Make cursor-open with next_random configured fail for column-store
objects, waiting until the next method is called seems unkind.
Rework the static random-cursor tests to smoke-test both forms of random
cursors.
- "join:" URI handling added to WT_SESSION::open_cursor, added WT_SESSION::join.
- Added support for in-memory bloom filters.
- Minor fix to cur_dump_close CURSOR_API_CALL.
- Some refactoring in __wt_{curindex,curtable}_get_value() to allow them
to be called internally.
- Use new macro JOINABLE_CURSOR_API_CALL() instead of CURSOR_API_CALL()
to trap illegal calls to joined cursor.
- __wt_curtable_open now takes an owning cursor arg.
- Added __wt_struct_unpack_size() and __wt_struct_repack()
I was seeing internal pages grow to contain over 70,000 entries. Which
makes reconciliation of child pages very expensive. Be more aggressive
about splitting internal pages.
