Files
mongo/examples/c/queue_example.h
Alex Gorrod b217c497e3 WT-2552 Add public API for pluggable filesystems (#2671)
* WT-2552 Add public API for pluggable filesystems

Not yet compiling. The main parts of this change should be here,
but it involved extensive parameter re-organization. There are also
a number of layering violations between our existing file system
implementations and the WT_FH, that aren't possible with the new
structure.

There are a number of specific todo comments in the code. One of the main
issues is that the in-memory file system had a special close semantic
that relied on WiredTiger handle tracking. The in-memory file-system should
do it's own tracking of file handles, I've gone part way down that road by
adding a queue for closed handles. Need to also add in live handles, and
manage the queue as appropriate.

I haven't created an example application that uses the new API yet.

* WT-2552 Add public API for pluggable filesystems

I always forget you have to remove the already-built html files when
changing PREDEFINED, add a reminder to the complaint.

* WT-2552 Add public API for pluggable filesystems

You have to remove the .js files, too.

* WT-2552 Add public API for pluggable filesystems

Make dist/s_all run cleanly.

* WT-2552 Add public API for pluggable filesystems

Whitespace.

* WT-2552 Add public API for pluggable filesystems

Make it compile/build/lint.

* WT-2552 Add public API for pluggable filesystems

block_write.c: In function '__wt_block_extend':
block_write.c:130:71: error: missing terminating ' character [-Werror]

* WT-2552 Add public API for pluggable filesystems

os_fs_inmemory.c: In function '__im_file_truncate':
os_fs_inmemory.c:344:10: error: 'session' is used uninitialized in this
function [-Werror=uninitialized]

* WT-2552 Add public API for pluggable filesystems

os_fs.c: In function '__posix_directory_sync':
os_fs.c:92:10: error: 'session' is used uninitialized in this function
[-Werror=uninitialized]

* WT-2552 Add public API for pluggable filesystems

Go back to using bool types in the file-system API, this requires we add
<stdbool.h> to the "standard" wiredtiger.h includes.

Consistently use wt_session to represent a WT_SESSION, we were using
"wtsession" in some places.

Make a pass over the Windows code, but I'm sure it doesn't compile yet.

* WT-2552 Add public API for pluggable filesystems

Fix up another couple of bool types.

* WT-2552 Add public API for pluggable filesystems

Move the file naming work out of the underlying filesystem functions,
the calls to __wt_filename are now in the upper-level code,n os_fs.i;
that means the filesystem code is no longer responsible for figuring out
paths. This is cleaner, although the directory-sync call is a bit of a
kluge, and I've commimtted us to handling NULL filesystem methods.

With this set of changes, in-memory runs again.

More Windows naming fixes.

* WT-2552 Add public API for pluggable filesystems

os_fs.c: In function '__posix_directory_sync':
os_fs.c:96:3: error: label 'err' used but not defined

* WT-2552 Add public API for pluggable filesystems

Pull out another call to __wt_filename() from the filesystem-dependent
code.

* WT-2552 Add public API for pluggable filesystems

Consistently check for missing file-system methods when doing
file-system calls.

Other minor lint & cleanup.

* WT-2552 Add public API for pluggable filesystems

Change the in-memory code to maintain a complete list of the files it
has ever opened, and depend on that list instead of reaching up into the
common layer for the WT_FH handle list.

This means __wt_handle_search is only used by the common WT_FH handle
code, simplify it, and add a __wt_handle_is_open function that can be
called for diagnostic purposes (to check for open files that are being
renamed or removed, for example).

* Fix comiler warning and ignore the file system API in Java

* Flesh out the example file system implementation.

* Add in some plumbing for set_file_system in wiredtiger_open.

* WT-2552 Add public API for pluggable filesystems

Whitespace.

* WT-2552 Add public API for pluggable filesystems

WT_CONFIG_ITEM.val isn't a boolean, don't use boolean types in
equal/not-equal comparisons.

* WT-2552 Add public API for pluggable filesystems

Remove unused #includes.

Increment/decrement the DEMO_FILE_SYSTEM.{opened,closed}_file_count.

Allocate demo structures, they're larger than the underlying structures.
Swap the number/size calloc arguments, number comes first.

Fix a couple of statics.

* WT-2552 Add public API for pluggable filesystems

Use %u instead of casting to %d.

* WT-2552 Add public API for pluggable filesystems

Add ex_file_system.c to the list of example programs.

* WT-2552 Add public API for pluggable filesystems

Change ex_file_system.c to not require <wt_internal.h>: strip down a
copy of FreeBSD's <queue.h> for local inclusion, rewrite a few other
minor pieces of code.

* WT-2552 Add public API for pluggable filesystems

Update spell check info

* WT-2552 Add public API for pluggable filesystems

__conn_load_extensions() shouldn't set the "early" boolean to true.

* WT-2552 Add public API for pluggable filesystems

Don't indirect through a NULL pointer if "local" was set and no path was
specified, always set the name to something useful.

* WT-2552 Add public API for pluggable filesystems

Don't indirect through a NULL pointer if "local" was set and no path was
specified, always set the name to something useful.

* WT-2552 Add public API for pluggable filesystems

wt_off_t vs. size_t conversion lint.

* WT-2552 Add public API for pluggable filesystems

Add -rdynamic to the load for ex_file_system, the main executable
symbols are not exported by default.

* WT-2552 Add public API for pluggable filesystems

The underlying handle name includes the enclosing directory,
compare against the WT_FH.name field instead.

* WT-2552 Add public API for pluggable filesystems

demo_fs_rename should return 0 if successful, simplify error handling

Don't bother casting arguments to free(), it's not necessary.

* WT-2552 Add public API for pluggable filesystems

General WT_FILE_SYSTEM cleanup.

Move OS initialization into the wiredtiger_open() code (the
os_common/os_init.c file is no longer needed).

Allow early-load extensions to be part of the environment settings,
matching the "in-memory" and "readonly" configurations.

Syntax check the set of a file-system, remove tests for NULL methods in
the file-system structure unless it's legal for them to be NULL.

Windows, POSIX and in-memory file systems now set WT_FILE_SYSTEM.terminate,
call that function to cleanup when discarding a WT_CONNECTION.

Export file-type and open-flags constants for WT_FILE_SYSTEM.open_file,
sort the WT_FILE_SYSTEM methods, do an editing pass.

Change the WT_FILE_HANDLE type from (const char *) to (char *), it's
"owned" by the underlying layer, and it's simpler that way.

Minor (untested) cleanup of the Windows WT_FILE_SYSTEM.open-file method.

* WT-2552 Add public API for pluggable filesystems

Export the advise argument #defines for the WT_FILE_HANDLE.fadvise method.

Sort the WT_FILE_HANDLE methods.

* WT-2552 Add public API for pluggable filesystems

Clean up and simplify WT_FILE_SYSTEM/WT_FILE_HANDLE documentation's
description of the handles.

* WT-2552 Add public API for pluggable filesystems

WT_FILE_HANDLE.close is a required function (at the least, it
has to free the memory).

WT_FILE_HANDLE.fadvise isn't a required function, if it's not
configured, don't call it.

* WT-2552 Add public API for pluggable filesystems

The WT_FILE_HANDLE.lock function is required.

Change the __wt_open() signature to match WT_FILE_SYSTEM.open_file().

* WT-2552 Add public API for pluggable filesystems

Rework all of the WT_FILE_HANDLE mapped region methods to be optional.

* WT-2552 Add public API for pluggable filesystems

The WT_FILE_HANDLE.{read,size} methods are required.
The WT_FILE_HANDLE.sync method is not required.

Split the WT_FILE_HANDLE.sync method into .sync and .sync_nowait versions,
it makes the upper-level code simpler (Windows supports .sync but doesn't
support .sync_nowait).

* WT-2552 Add public API for pluggable filesystems

The WT_FILE_HANDLE.{truncate,write} methods are required IFF the file
is not readonly.

* WT-2552 Add public API for pluggable filesystems

POSIX shouldn't declare a no-sync handle function unless the
sync_file_range system call is available.

* WT-2552 Add public API for pluggable filesystems

Typo, missing semi-colon.

* Fix a bug in ex_file_system.c

* Fix a memory leak in posix file handle implementation

* WT-2552 Use the correct flags when opening backup file.

* WT-2552 Add public API for pluggable filesystems

Simplify open-file error handling by calling the close function on the
handle, that way we won't forget to free all of the applicable memory
allocations.

* WT-2552 Add public API for pluggable filesystems

Simplify the directory-list method, don't pass in an include/exclude
file, if prefix is non-NULL, it implies we only want files matching
the prefix.

* WT-2552 Add public API for pluggable filesystems

Replace WT_FILE_HANDLE_POSIX.fallocate_{available,requires_locking} wiht
WT_FILE_HANDLE.fallocate and WT_FILE_HANDLE.fallocate_nolock.

Example code doesn't need to set WT_FILE_HANDLE methods to NULL, the
allocation does that.

Free the I/O buffer if open-handle allocation fails in the example code.

Remove snippets for WT_FILE_SYSTEM and WT_FILE_HANDLE methods, we're
not going to provide example code for them.

* WT-2552 Add public API for pluggable filesystems

Document we expect either ENOTSUP or EBUSY from optionally supported
APIs. Review/cleanups ENOTSUP/EBUSY returns from optionally supported
APIs.

Make WT_FILE_HANDLE.lock optional.

Don't configure or call the POSIX fadvise function on files configured
for direct I/O.

Rename __wt_filesize_name to __wt_size for consistency.

Update the spelling list.

* WT-2552 Add public API for pluggable filesystems

WT_FILE_HANDLE.truncate requires locking in all known implementations,
document it is not called concurrently with other operations.

* WT-2552 Add public API for pluggable filesystems

Don't terminate the filesystem unless we've actually configured one.

* WT-2552 Add public API for pluggable filesystems

Remove WT_FILE_SYSTEM and WT_FILE_HANDLE from SWIG so the test suite
can pass again.

* WT-2552 Add public API for pluggable filesystems

Merge __conn_load_early_extensions() and __conn_load_extensions().

Fix a problem where I moved the early extensions load to where it could
include the WiredTiger environment variable, but I didn't pass the built
cfg into the function.

* WT-2552 Add public API for pluggable filesystems

Linux build typo.

* WT-2552 Add public API for pluggable filesystems

Get rid of the "bool silent" argument to WT_FILE_SYSTEM.size by testing
for the file's existence before requesting the size (an extra system
call, but guaranteed to hit in the buffer cache at least).

* WT-2552 Add public API for pluggable filesystems

Naming consistency pass over the WT_FILE_SYSTEM functions.

* WT-2552 Add public API for pluggable filesystems

Fix a spin lock mismatch.

* WT-2552 Add public API for pluggable filesystems

Another spinlock mismatch.

* Update example pluggable file system.

Add a directory list implementation to the example, which uncovered
an issue with the API. The directory list API allocates memory that
is freed by WiredTiger, which I don't think is kosher.

* Change file-directory-sync to use reguar fsync.

The distinction in os_fs.i doesn't work with the filesystem API.

Also add directory_sync application to the example application.

* WT-2552 Add public API for pluggable filesystems

Whitespace.

* WT-2552 Add public API for pluggable filesystems

Rewrite __wt_free to not evaluate macro arguments multiple times.

* WT-2552 Add public API for pluggable filesystems

Simplify the directory-list functions: __wt_realloc_def() already
handles scaling the size of the allocations, there's no need to
involve a separate constant that increments the allocation size.

* WT-2552 Add public API for pluggable filesystems

Fix a grouping problem in a realloc call, we need to multiple the size
times the previously allocated slots + 10.

Fix buffer overrun, if "count" has already been incremented, the memset
would skip clearing the first slot and clear one slot past the end of
the buffer.

Remove a comment, realloc requires clearing allocated memory, it's not
paranoia.

* WT-2552 Add public API for pluggable filesystems

Add the mapping-cookie argument to the map-preload and map-discard
functions.

Change page-discard to stop reaching down through the block manager,
instead, provide a block-manager map-discard function that does the
work.

* WT-2552 Add public API for pluggable filesystems

Require a directory-list function.

Implement a directory-list function for the in-memory filesystem.

Consistency pass, make all the directory-list functions look the same.

* WT-2552 Add public API for pluggable filesystems

The WT_FILE_SYSTEM.{directory_sync, remove, rename} methods are not
required for read-only systems.

* WT-2552 Add public API for pluggable filesystems

Change the WT_FILE_SYSTEM.open_file file_type argument from a set of
constants to an enum.

This requires changing how we store connection direct I/O configuration
(the constants used to be flags stored in the WT_CONNECTION_IMPL), and
requiring all callers of __wt_open() do their own work to figure out if
WT_OPEN_DIRECTIO should be specified.

* WT-2552 Add public API for pluggable filesystems

Make no guarantees WT_FILE_SYSTEM and WT_FILE_HANDLE methods are
not called concurrently (except for WT_FILE_HANDLE::fallocate and
WT_FILE_HANDLE::fallocate_nolock).

Rewrite the in-memory FS code to lock across all methods (for example,
WT_FILE_HANDLE.close), that means including a reference to the enclosing
WT_FILE_SYSTEM in the WT_FILE_HANDLE structure so we can find a lock
without using the WT_CONNECTION_IMPL structure.

* WT-2552 Add public API for pluggable filesystems

Remove __wt_directory_sync_fh, it's no longer useful.

* WT-2552 Add public API for pluggable filesystems

Rename WT_INMEMORY_FILE_SYSTEM to WT_FILE_SYSTEM_INMEM, matching
WT_FILE_HANDLE_INMEM.

* WT-2552 Add public API for pluggable filesystems

Add WT_FILE_SYSTEM.directory_list_free, to free memory allocated
by WT_FILE_SYSTEM.direct_list.

Fix a memory leak in __log_archive_once (if __wt_readlock failed,
we leaked the directory-list memory).

* WT-2552 Add public API for pluggable filesystems

Typo, check WT_DIRECT_IO_LOG, not WT_DIRECT_IO_CHECKPOINT.

* WT-2552 Add public API for pluggable filesystems

Typo, unreachable code.

* WT-2552 Add public API for pluggable filesystems

We don't require WT_FILE_SYSTEM.{remove,rename} if the system is
read-only.

* Fix Windows build with pluggable file system.

Involved removing u_int from the public API.

* Fix line wrapping.

* Fix Windows terminate function.

* Forgot something in my last commit.

* Fix Windows munmap bug.

* Add new example to Windows build. Extend example to be more complete.

* Fix example loading on Windows

* Update documentation

* Add missing spell words

* Remove old comment.
2016-04-28 07:16:44 -04:00

150 lines
5.7 KiB
C

/*-
* Public Domain 2014-2016 MongoDB, Inc.
* Public Domain 2008-2014 WiredTiger, Inc.
*
* This is free and unencumbered software released into the public domain.
*
* Anyone is free to copy, modify, publish, use, compile, sell, or
* distribute this software, either in source code form or as a compiled
* binary, for any purpose, commercial or non-commercial, and by any
* means.
*
* In jurisdictions that recognize copyright laws, the author or authors
* of this software dedicate any and all copyright interest in the
* software to the public domain. We make this dedication for the benefit
* of the public at large and to the detriment of our heirs and
* successors. We intend this dedication to be an overt act of
* relinquishment in perpetuity of all present and future rights to this
* software under copyright law.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
* OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
* ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
* OTHER DEALINGS IN THE SOFTWARE.
*/
/*
* This is a stripped down copy of the FreeBSD queue.h include file to make
* TAILQ_XXX functionality available in WiredTiger example programs.
*/
/*-
* Copyright (c) 1991, 1993
* The Regents of the University of California. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 4. Neither the name of the University nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
* @(#)queue.h 8.5 (Berkeley) 8/20/94
* $FreeBSD: releng/10.2/sys/sys/queue.h 279633 2015-03-05 09:23:43Z hselasky $
*/
/*
* Tail queue declarations.
*/
#define TAILQ_HEAD(name, type) \
struct name { \
struct type *tqh_first; /* first element */ \
struct type **tqh_last; /* addr of last next element */ \
}
#define TAILQ_ENTRY(type) \
struct { \
struct type *tqe_next; /* next element */ \
struct type **tqe_prev; /* address of previous next element */ \
}
/*
* Tail queue functions.
*/
#define TAILQ_EMPTY(head) ((head)->tqh_first == NULL)
#define TAILQ_FIRST(head) ((head)->tqh_first)
#define TAILQ_FOREACH(var, head, field) \
for ((var) = TAILQ_FIRST((head)); \
(var); \
(var) = TAILQ_NEXT((var), field))
#define TAILQ_INIT(head) do { \
TAILQ_FIRST((head)) = NULL; \
(head)->tqh_last = &TAILQ_FIRST((head)); \
} while (0)
#define TAILQ_INSERT_AFTER(head, listelm, elm, field) do { \
if ((TAILQ_NEXT((elm), field) = TAILQ_NEXT((listelm), field)) != NULL)\
TAILQ_NEXT((elm), field)->field.tqe_prev = \
&TAILQ_NEXT((elm), field); \
else { \
(head)->tqh_last = &TAILQ_NEXT((elm), field); \
} \
TAILQ_NEXT((listelm), field) = (elm); \
(elm)->field.tqe_prev = &TAILQ_NEXT((listelm), field); \
} while (0)
#define TAILQ_INSERT_BEFORE(listelm, elm, field) do { \
(elm)->field.tqe_prev = (listelm)->field.tqe_prev; \
TAILQ_NEXT((elm), field) = (listelm); \
*(listelm)->field.tqe_prev = (elm); \
(listelm)->field.tqe_prev = &TAILQ_NEXT((elm), field); \
} while (0)
#define TAILQ_INSERT_HEAD(head, elm, field) do { \
if ((TAILQ_NEXT((elm), field) = TAILQ_FIRST((head))) != NULL) \
TAILQ_FIRST((head))->field.tqe_prev = \
&TAILQ_NEXT((elm), field); \
else \
(head)->tqh_last = &TAILQ_NEXT((elm), field); \
TAILQ_FIRST((head)) = (elm); \
(elm)->field.tqe_prev = &TAILQ_FIRST((head)); \
} while (0)
#define TAILQ_INSERT_TAIL(head, elm, field) do { \
TAILQ_NEXT((elm), field) = NULL; \
(elm)->field.tqe_prev = (head)->tqh_last; \
*(head)->tqh_last = (elm); \
(head)->tqh_last = &TAILQ_NEXT((elm), field); \
} while (0)
#define TAILQ_LAST(head, headname) \
(*(((struct headname *)((head)->tqh_last))->tqh_last))
#define TAILQ_NEXT(elm, field) ((elm)->field.tqe_next)
#define TAILQ_PREV(elm, headname, field) \
(*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
#define TAILQ_REMOVE(head, elm, field) do { \
if ((TAILQ_NEXT((elm), field)) != NULL) \
TAILQ_NEXT((elm), field)->field.tqe_prev = \
(elm)->field.tqe_prev; \
else { \
(head)->tqh_last = (elm)->field.tqe_prev; \
} \
*(elm)->field.tqe_prev = TAILQ_NEXT((elm), field); \
} while (0)