Files
mongo/examples/c/queue_example.h

150 lines
5.7 KiB
C
Raw Normal View History

WT-2552 Add public API for pluggable filesystems (#2671) * WT-2552 Add public API for pluggable filesystems Not yet compiling. The main parts of this change should be here, but it involved extensive parameter re-organization. There are also a number of layering violations between our existing file system implementations and the WT_FH, that aren't possible with the new structure. There are a number of specific todo comments in the code. One of the main issues is that the in-memory file system had a special close semantic that relied on WiredTiger handle tracking. The in-memory file-system should do it's own tracking of file handles, I've gone part way down that road by adding a queue for closed handles. Need to also add in live handles, and manage the queue as appropriate. I haven't created an example application that uses the new API yet. * WT-2552 Add public API for pluggable filesystems I always forget you have to remove the already-built html files when changing PREDEFINED, add a reminder to the complaint. * WT-2552 Add public API for pluggable filesystems You have to remove the .js files, too. * WT-2552 Add public API for pluggable filesystems Make dist/s_all run cleanly. * WT-2552 Add public API for pluggable filesystems Whitespace. * WT-2552 Add public API for pluggable filesystems Make it compile/build/lint. * WT-2552 Add public API for pluggable filesystems block_write.c: In function '__wt_block_extend': block_write.c:130:71: error: missing terminating ' character [-Werror] * WT-2552 Add public API for pluggable filesystems os_fs_inmemory.c: In function '__im_file_truncate': os_fs_inmemory.c:344:10: error: 'session' is used uninitialized in this function [-Werror=uninitialized] * WT-2552 Add public API for pluggable filesystems os_fs.c: In function '__posix_directory_sync': os_fs.c:92:10: error: 'session' is used uninitialized in this function [-Werror=uninitialized] * WT-2552 Add public API for pluggable filesystems Go back to using bool types in the file-system API, this requires we add <stdbool.h> to the "standard" wiredtiger.h includes. Consistently use wt_session to represent a WT_SESSION, we were using "wtsession" in some places. Make a pass over the Windows code, but I'm sure it doesn't compile yet. * WT-2552 Add public API for pluggable filesystems Fix up another couple of bool types. * WT-2552 Add public API for pluggable filesystems Move the file naming work out of the underlying filesystem functions, the calls to __wt_filename are now in the upper-level code,n os_fs.i; that means the filesystem code is no longer responsible for figuring out paths. This is cleaner, although the directory-sync call is a bit of a kluge, and I've commimtted us to handling NULL filesystem methods. With this set of changes, in-memory runs again. More Windows naming fixes. * WT-2552 Add public API for pluggable filesystems os_fs.c: In function '__posix_directory_sync': os_fs.c:96:3: error: label 'err' used but not defined * WT-2552 Add public API for pluggable filesystems Pull out another call to __wt_filename() from the filesystem-dependent code. * WT-2552 Add public API for pluggable filesystems Consistently check for missing file-system methods when doing file-system calls. Other minor lint & cleanup. * WT-2552 Add public API for pluggable filesystems Change the in-memory code to maintain a complete list of the files it has ever opened, and depend on that list instead of reaching up into the common layer for the WT_FH handle list. This means __wt_handle_search is only used by the common WT_FH handle code, simplify it, and add a __wt_handle_is_open function that can be called for diagnostic purposes (to check for open files that are being renamed or removed, for example). * Fix comiler warning and ignore the file system API in Java * Flesh out the example file system implementation. * Add in some plumbing for set_file_system in wiredtiger_open. * WT-2552 Add public API for pluggable filesystems Whitespace. * WT-2552 Add public API for pluggable filesystems WT_CONFIG_ITEM.val isn't a boolean, don't use boolean types in equal/not-equal comparisons. * WT-2552 Add public API for pluggable filesystems Remove unused #includes. Increment/decrement the DEMO_FILE_SYSTEM.{opened,closed}_file_count. Allocate demo structures, they're larger than the underlying structures. Swap the number/size calloc arguments, number comes first. Fix a couple of statics. * WT-2552 Add public API for pluggable filesystems Use %u instead of casting to %d. * WT-2552 Add public API for pluggable filesystems Add ex_file_system.c to the list of example programs. * WT-2552 Add public API for pluggable filesystems Change ex_file_system.c to not require <wt_internal.h>: strip down a copy of FreeBSD's <queue.h> for local inclusion, rewrite a few other minor pieces of code. * WT-2552 Add public API for pluggable filesystems Update spell check info * WT-2552 Add public API for pluggable filesystems __conn_load_extensions() shouldn't set the "early" boolean to true. * WT-2552 Add public API for pluggable filesystems Don't indirect through a NULL pointer if "local" was set and no path was specified, always set the name to something useful. * WT-2552 Add public API for pluggable filesystems Don't indirect through a NULL pointer if "local" was set and no path was specified, always set the name to something useful. * WT-2552 Add public API for pluggable filesystems wt_off_t vs. size_t conversion lint. * WT-2552 Add public API for pluggable filesystems Add -rdynamic to the load for ex_file_system, the main executable symbols are not exported by default. * WT-2552 Add public API for pluggable filesystems The underlying handle name includes the enclosing directory, compare against the WT_FH.name field instead. * WT-2552 Add public API for pluggable filesystems demo_fs_rename should return 0 if successful, simplify error handling Don't bother casting arguments to free(), it's not necessary. * WT-2552 Add public API for pluggable filesystems General WT_FILE_SYSTEM cleanup. Move OS initialization into the wiredtiger_open() code (the os_common/os_init.c file is no longer needed). Allow early-load extensions to be part of the environment settings, matching the "in-memory" and "readonly" configurations. Syntax check the set of a file-system, remove tests for NULL methods in the file-system structure unless it's legal for them to be NULL. Windows, POSIX and in-memory file systems now set WT_FILE_SYSTEM.terminate, call that function to cleanup when discarding a WT_CONNECTION. Export file-type and open-flags constants for WT_FILE_SYSTEM.open_file, sort the WT_FILE_SYSTEM methods, do an editing pass. Change the WT_FILE_HANDLE type from (const char *) to (char *), it's "owned" by the underlying layer, and it's simpler that way. Minor (untested) cleanup of the Windows WT_FILE_SYSTEM.open-file method. * WT-2552 Add public API for pluggable filesystems Export the advise argument #defines for the WT_FILE_HANDLE.fadvise method. Sort the WT_FILE_HANDLE methods. * WT-2552 Add public API for pluggable filesystems Clean up and simplify WT_FILE_SYSTEM/WT_FILE_HANDLE documentation's description of the handles. * WT-2552 Add public API for pluggable filesystems WT_FILE_HANDLE.close is a required function (at the least, it has to free the memory). WT_FILE_HANDLE.fadvise isn't a required function, if it's not configured, don't call it. * WT-2552 Add public API for pluggable filesystems The WT_FILE_HANDLE.lock function is required. Change the __wt_open() signature to match WT_FILE_SYSTEM.open_file(). * WT-2552 Add public API for pluggable filesystems Rework all of the WT_FILE_HANDLE mapped region methods to be optional. * WT-2552 Add public API for pluggable filesystems The WT_FILE_HANDLE.{read,size} methods are required. The WT_FILE_HANDLE.sync method is not required. Split the WT_FILE_HANDLE.sync method into .sync and .sync_nowait versions, it makes the upper-level code simpler (Windows supports .sync but doesn't support .sync_nowait). * WT-2552 Add public API for pluggable filesystems The WT_FILE_HANDLE.{truncate,write} methods are required IFF the file is not readonly. * WT-2552 Add public API for pluggable filesystems POSIX shouldn't declare a no-sync handle function unless the sync_file_range system call is available. * WT-2552 Add public API for pluggable filesystems Typo, missing semi-colon. * Fix a bug in ex_file_system.c * Fix a memory leak in posix file handle implementation * WT-2552 Use the correct flags when opening backup file. * WT-2552 Add public API for pluggable filesystems Simplify open-file error handling by calling the close function on the handle, that way we won't forget to free all of the applicable memory allocations. * WT-2552 Add public API for pluggable filesystems Simplify the directory-list method, don't pass in an include/exclude file, if prefix is non-NULL, it implies we only want files matching the prefix. * WT-2552 Add public API for pluggable filesystems Replace WT_FILE_HANDLE_POSIX.fallocate_{available,requires_locking} wiht WT_FILE_HANDLE.fallocate and WT_FILE_HANDLE.fallocate_nolock. Example code doesn't need to set WT_FILE_HANDLE methods to NULL, the allocation does that. Free the I/O buffer if open-handle allocation fails in the example code. Remove snippets for WT_FILE_SYSTEM and WT_FILE_HANDLE methods, we're not going to provide example code for them. * WT-2552 Add public API for pluggable filesystems Document we expect either ENOTSUP or EBUSY from optionally supported APIs. Review/cleanups ENOTSUP/EBUSY returns from optionally supported APIs. Make WT_FILE_HANDLE.lock optional. Don't configure or call the POSIX fadvise function on files configured for direct I/O. Rename __wt_filesize_name to __wt_size for consistency. Update the spelling list. * WT-2552 Add public API for pluggable filesystems WT_FILE_HANDLE.truncate requires locking in all known implementations, document it is not called concurrently with other operations. * WT-2552 Add public API for pluggable filesystems Don't terminate the filesystem unless we've actually configured one. * WT-2552 Add public API for pluggable filesystems Remove WT_FILE_SYSTEM and WT_FILE_HANDLE from SWIG so the test suite can pass again. * WT-2552 Add public API for pluggable filesystems Merge __conn_load_early_extensions() and __conn_load_extensions(). Fix a problem where I moved the early extensions load to where it could include the WiredTiger environment variable, but I didn't pass the built cfg into the function. * WT-2552 Add public API for pluggable filesystems Linux build typo. * WT-2552 Add public API for pluggable filesystems Get rid of the "bool silent" argument to WT_FILE_SYSTEM.size by testing for the file's existence before requesting the size (an extra system call, but guaranteed to hit in the buffer cache at least). * WT-2552 Add public API for pluggable filesystems Naming consistency pass over the WT_FILE_SYSTEM functions. * WT-2552 Add public API for pluggable filesystems Fix a spin lock mismatch. * WT-2552 Add public API for pluggable filesystems Another spinlock mismatch. * Update example pluggable file system. Add a directory list implementation to the example, which uncovered an issue with the API. The directory list API allocates memory that is freed by WiredTiger, which I don't think is kosher. * Change file-directory-sync to use reguar fsync. The distinction in os_fs.i doesn't work with the filesystem API. Also add directory_sync application to the example application. * WT-2552 Add public API for pluggable filesystems Whitespace. * WT-2552 Add public API for pluggable filesystems Rewrite __wt_free to not evaluate macro arguments multiple times. * WT-2552 Add public API for pluggable filesystems Simplify the directory-list functions: __wt_realloc_def() already handles scaling the size of the allocations, there's no need to involve a separate constant that increments the allocation size. * WT-2552 Add public API for pluggable filesystems Fix a grouping problem in a realloc call, we need to multiple the size times the previously allocated slots + 10. Fix buffer overrun, if "count" has already been incremented, the memset would skip clearing the first slot and clear one slot past the end of the buffer. Remove a comment, realloc requires clearing allocated memory, it's not paranoia. * WT-2552 Add public API for pluggable filesystems Add the mapping-cookie argument to the map-preload and map-discard functions. Change page-discard to stop reaching down through the block manager, instead, provide a block-manager map-discard function that does the work. * WT-2552 Add public API for pluggable filesystems Require a directory-list function. Implement a directory-list function for the in-memory filesystem. Consistency pass, make all the directory-list functions look the same. * WT-2552 Add public API for pluggable filesystems The WT_FILE_SYSTEM.{directory_sync, remove, rename} methods are not required for read-only systems. * WT-2552 Add public API for pluggable filesystems Change the WT_FILE_SYSTEM.open_file file_type argument from a set of constants to an enum. This requires changing how we store connection direct I/O configuration (the constants used to be flags stored in the WT_CONNECTION_IMPL), and requiring all callers of __wt_open() do their own work to figure out if WT_OPEN_DIRECTIO should be specified. * WT-2552 Add public API for pluggable filesystems Make no guarantees WT_FILE_SYSTEM and WT_FILE_HANDLE methods are not called concurrently (except for WT_FILE_HANDLE::fallocate and WT_FILE_HANDLE::fallocate_nolock). Rewrite the in-memory FS code to lock across all methods (for example, WT_FILE_HANDLE.close), that means including a reference to the enclosing WT_FILE_SYSTEM in the WT_FILE_HANDLE structure so we can find a lock without using the WT_CONNECTION_IMPL structure. * WT-2552 Add public API for pluggable filesystems Remove __wt_directory_sync_fh, it's no longer useful. * WT-2552 Add public API for pluggable filesystems Rename WT_INMEMORY_FILE_SYSTEM to WT_FILE_SYSTEM_INMEM, matching WT_FILE_HANDLE_INMEM. * WT-2552 Add public API for pluggable filesystems Add WT_FILE_SYSTEM.directory_list_free, to free memory allocated by WT_FILE_SYSTEM.direct_list. Fix a memory leak in __log_archive_once (if __wt_readlock failed, we leaked the directory-list memory). * WT-2552 Add public API for pluggable filesystems Typo, check WT_DIRECT_IO_LOG, not WT_DIRECT_IO_CHECKPOINT. * WT-2552 Add public API for pluggable filesystems Typo, unreachable code. * WT-2552 Add public API for pluggable filesystems We don't require WT_FILE_SYSTEM.{remove,rename} if the system is read-only. * Fix Windows build with pluggable file system. Involved removing u_int from the public API. * Fix line wrapping. * Fix Windows terminate function. * Forgot something in my last commit. * Fix Windows munmap bug. * Add new example to Windows build. Extend example to be more complete. * Fix example loading on Windows * Update documentation * Add missing spell words * Remove old comment.
2016-04-28 21:16:44 +10:00
/*-
* Public Domain 2014-2016 MongoDB, Inc.
* Public Domain 2008-2014 WiredTiger, Inc.
*
* This is free and unencumbered software released into the public domain.
*
* Anyone is free to copy, modify, publish, use, compile, sell, or
* distribute this software, either in source code form or as a compiled
* binary, for any purpose, commercial or non-commercial, and by any
* means.
*
* In jurisdictions that recognize copyright laws, the author or authors
* of this software dedicate any and all copyright interest in the
* software to the public domain. We make this dedication for the benefit
* of the public at large and to the detriment of our heirs and
* successors. We intend this dedication to be an overt act of
* relinquishment in perpetuity of all present and future rights to this
* software under copyright law.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
* OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
* ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
* OTHER DEALINGS IN THE SOFTWARE.
*/
/*
* This is a stripped down copy of the FreeBSD queue.h include file to make
* TAILQ_XXX functionality available in WiredTiger example programs.
*/
/*-
* Copyright (c) 1991, 1993
* The Regents of the University of California. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 4. Neither the name of the University nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
* @(#)queue.h 8.5 (Berkeley) 8/20/94
* $FreeBSD: releng/10.2/sys/sys/queue.h 279633 2015-03-05 09:23:43Z hselasky $
*/
/*
* Tail queue declarations.
*/
#define TAILQ_HEAD(name, type) \
struct name { \
struct type *tqh_first; /* first element */ \
struct type **tqh_last; /* addr of last next element */ \
}
#define TAILQ_ENTRY(type) \
struct { \
struct type *tqe_next; /* next element */ \
struct type **tqe_prev; /* address of previous next element */ \
}
/*
* Tail queue functions.
*/
#define TAILQ_EMPTY(head) ((head)->tqh_first == NULL)
#define TAILQ_FIRST(head) ((head)->tqh_first)
#define TAILQ_FOREACH(var, head, field) \
for ((var) = TAILQ_FIRST((head)); \
(var); \
(var) = TAILQ_NEXT((var), field))
#define TAILQ_INIT(head) do { \
TAILQ_FIRST((head)) = NULL; \
(head)->tqh_last = &TAILQ_FIRST((head)); \
} while (0)
#define TAILQ_INSERT_AFTER(head, listelm, elm, field) do { \
if ((TAILQ_NEXT((elm), field) = TAILQ_NEXT((listelm), field)) != NULL)\
TAILQ_NEXT((elm), field)->field.tqe_prev = \
&TAILQ_NEXT((elm), field); \
else { \
(head)->tqh_last = &TAILQ_NEXT((elm), field); \
} \
TAILQ_NEXT((listelm), field) = (elm); \
(elm)->field.tqe_prev = &TAILQ_NEXT((listelm), field); \
} while (0)
#define TAILQ_INSERT_BEFORE(listelm, elm, field) do { \
(elm)->field.tqe_prev = (listelm)->field.tqe_prev; \
TAILQ_NEXT((elm), field) = (listelm); \
*(listelm)->field.tqe_prev = (elm); \
(listelm)->field.tqe_prev = &TAILQ_NEXT((elm), field); \
} while (0)
#define TAILQ_INSERT_HEAD(head, elm, field) do { \
if ((TAILQ_NEXT((elm), field) = TAILQ_FIRST((head))) != NULL) \
TAILQ_FIRST((head))->field.tqe_prev = \
&TAILQ_NEXT((elm), field); \
else \
(head)->tqh_last = &TAILQ_NEXT((elm), field); \
TAILQ_FIRST((head)) = (elm); \
(elm)->field.tqe_prev = &TAILQ_FIRST((head)); \
} while (0)
#define TAILQ_INSERT_TAIL(head, elm, field) do { \
TAILQ_NEXT((elm), field) = NULL; \
(elm)->field.tqe_prev = (head)->tqh_last; \
*(head)->tqh_last = (elm); \
(head)->tqh_last = &TAILQ_NEXT((elm), field); \
} while (0)
#define TAILQ_LAST(head, headname) \
(*(((struct headname *)((head)->tqh_last))->tqh_last))
#define TAILQ_NEXT(elm, field) ((elm)->field.tqe_next)
#define TAILQ_PREV(elm, headname, field) \
(*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
#define TAILQ_REMOVE(head, elm, field) do { \
if ((TAILQ_NEXT((elm), field)) != NULL) \
TAILQ_NEXT((elm), field)->field.tqe_prev = \
(elm)->field.tqe_prev; \
else { \
(head)->tqh_last = (elm)->field.tqe_prev; \
} \
*(elm)->field.tqe_prev = TAILQ_NEXT((elm), field); \
} while (0)