Files
mongo/dist/flags.py
Alex Gorrod b217c497e3 WT-2552 Add public API for pluggable filesystems (#2671)
* WT-2552 Add public API for pluggable filesystems

Not yet compiling. The main parts of this change should be here,
but it involved extensive parameter re-organization. There are also
a number of layering violations between our existing file system
implementations and the WT_FH, that aren't possible with the new
structure.

There are a number of specific todo comments in the code. One of the main
issues is that the in-memory file system had a special close semantic
that relied on WiredTiger handle tracking. The in-memory file-system should
do it's own tracking of file handles, I've gone part way down that road by
adding a queue for closed handles. Need to also add in live handles, and
manage the queue as appropriate.

I haven't created an example application that uses the new API yet.

* WT-2552 Add public API for pluggable filesystems

I always forget you have to remove the already-built html files when
changing PREDEFINED, add a reminder to the complaint.

* WT-2552 Add public API for pluggable filesystems

You have to remove the .js files, too.

* WT-2552 Add public API for pluggable filesystems

Make dist/s_all run cleanly.

* WT-2552 Add public API for pluggable filesystems

Whitespace.

* WT-2552 Add public API for pluggable filesystems

Make it compile/build/lint.

* WT-2552 Add public API for pluggable filesystems

block_write.c: In function '__wt_block_extend':
block_write.c:130:71: error: missing terminating ' character [-Werror]

* WT-2552 Add public API for pluggable filesystems

os_fs_inmemory.c: In function '__im_file_truncate':
os_fs_inmemory.c:344:10: error: 'session' is used uninitialized in this
function [-Werror=uninitialized]

* WT-2552 Add public API for pluggable filesystems

os_fs.c: In function '__posix_directory_sync':
os_fs.c:92:10: error: 'session' is used uninitialized in this function
[-Werror=uninitialized]

* WT-2552 Add public API for pluggable filesystems

Go back to using bool types in the file-system API, this requires we add
<stdbool.h> to the "standard" wiredtiger.h includes.

Consistently use wt_session to represent a WT_SESSION, we were using
"wtsession" in some places.

Make a pass over the Windows code, but I'm sure it doesn't compile yet.

* WT-2552 Add public API for pluggable filesystems

Fix up another couple of bool types.

* WT-2552 Add public API for pluggable filesystems

Move the file naming work out of the underlying filesystem functions,
the calls to __wt_filename are now in the upper-level code,n os_fs.i;
that means the filesystem code is no longer responsible for figuring out
paths. This is cleaner, although the directory-sync call is a bit of a
kluge, and I've commimtted us to handling NULL filesystem methods.

With this set of changes, in-memory runs again.

More Windows naming fixes.

* WT-2552 Add public API for pluggable filesystems

os_fs.c: In function '__posix_directory_sync':
os_fs.c:96:3: error: label 'err' used but not defined

* WT-2552 Add public API for pluggable filesystems

Pull out another call to __wt_filename() from the filesystem-dependent
code.

* WT-2552 Add public API for pluggable filesystems

Consistently check for missing file-system methods when doing
file-system calls.

Other minor lint & cleanup.

* WT-2552 Add public API for pluggable filesystems

Change the in-memory code to maintain a complete list of the files it
has ever opened, and depend on that list instead of reaching up into the
common layer for the WT_FH handle list.

This means __wt_handle_search is only used by the common WT_FH handle
code, simplify it, and add a __wt_handle_is_open function that can be
called for diagnostic purposes (to check for open files that are being
renamed or removed, for example).

* Fix comiler warning and ignore the file system API in Java

* Flesh out the example file system implementation.

* Add in some plumbing for set_file_system in wiredtiger_open.

* WT-2552 Add public API for pluggable filesystems

Whitespace.

* WT-2552 Add public API for pluggable filesystems

WT_CONFIG_ITEM.val isn't a boolean, don't use boolean types in
equal/not-equal comparisons.

* WT-2552 Add public API for pluggable filesystems

Remove unused #includes.

Increment/decrement the DEMO_FILE_SYSTEM.{opened,closed}_file_count.

Allocate demo structures, they're larger than the underlying structures.
Swap the number/size calloc arguments, number comes first.

Fix a couple of statics.

* WT-2552 Add public API for pluggable filesystems

Use %u instead of casting to %d.

* WT-2552 Add public API for pluggable filesystems

Add ex_file_system.c to the list of example programs.

* WT-2552 Add public API for pluggable filesystems

Change ex_file_system.c to not require <wt_internal.h>: strip down a
copy of FreeBSD's <queue.h> for local inclusion, rewrite a few other
minor pieces of code.

* WT-2552 Add public API for pluggable filesystems

Update spell check info

* WT-2552 Add public API for pluggable filesystems

__conn_load_extensions() shouldn't set the "early" boolean to true.

* WT-2552 Add public API for pluggable filesystems

Don't indirect through a NULL pointer if "local" was set and no path was
specified, always set the name to something useful.

* WT-2552 Add public API for pluggable filesystems

Don't indirect through a NULL pointer if "local" was set and no path was
specified, always set the name to something useful.

* WT-2552 Add public API for pluggable filesystems

wt_off_t vs. size_t conversion lint.

* WT-2552 Add public API for pluggable filesystems

Add -rdynamic to the load for ex_file_system, the main executable
symbols are not exported by default.

* WT-2552 Add public API for pluggable filesystems

The underlying handle name includes the enclosing directory,
compare against the WT_FH.name field instead.

* WT-2552 Add public API for pluggable filesystems

demo_fs_rename should return 0 if successful, simplify error handling

Don't bother casting arguments to free(), it's not necessary.

* WT-2552 Add public API for pluggable filesystems

General WT_FILE_SYSTEM cleanup.

Move OS initialization into the wiredtiger_open() code (the
os_common/os_init.c file is no longer needed).

Allow early-load extensions to be part of the environment settings,
matching the "in-memory" and "readonly" configurations.

Syntax check the set of a file-system, remove tests for NULL methods in
the file-system structure unless it's legal for them to be NULL.

Windows, POSIX and in-memory file systems now set WT_FILE_SYSTEM.terminate,
call that function to cleanup when discarding a WT_CONNECTION.

Export file-type and open-flags constants for WT_FILE_SYSTEM.open_file,
sort the WT_FILE_SYSTEM methods, do an editing pass.

Change the WT_FILE_HANDLE type from (const char *) to (char *), it's
"owned" by the underlying layer, and it's simpler that way.

Minor (untested) cleanup of the Windows WT_FILE_SYSTEM.open-file method.

* WT-2552 Add public API for pluggable filesystems

Export the advise argument #defines for the WT_FILE_HANDLE.fadvise method.

Sort the WT_FILE_HANDLE methods.

* WT-2552 Add public API for pluggable filesystems

Clean up and simplify WT_FILE_SYSTEM/WT_FILE_HANDLE documentation's
description of the handles.

* WT-2552 Add public API for pluggable filesystems

WT_FILE_HANDLE.close is a required function (at the least, it
has to free the memory).

WT_FILE_HANDLE.fadvise isn't a required function, if it's not
configured, don't call it.

* WT-2552 Add public API for pluggable filesystems

The WT_FILE_HANDLE.lock function is required.

Change the __wt_open() signature to match WT_FILE_SYSTEM.open_file().

* WT-2552 Add public API for pluggable filesystems

Rework all of the WT_FILE_HANDLE mapped region methods to be optional.

* WT-2552 Add public API for pluggable filesystems

The WT_FILE_HANDLE.{read,size} methods are required.
The WT_FILE_HANDLE.sync method is not required.

Split the WT_FILE_HANDLE.sync method into .sync and .sync_nowait versions,
it makes the upper-level code simpler (Windows supports .sync but doesn't
support .sync_nowait).

* WT-2552 Add public API for pluggable filesystems

The WT_FILE_HANDLE.{truncate,write} methods are required IFF the file
is not readonly.

* WT-2552 Add public API for pluggable filesystems

POSIX shouldn't declare a no-sync handle function unless the
sync_file_range system call is available.

* WT-2552 Add public API for pluggable filesystems

Typo, missing semi-colon.

* Fix a bug in ex_file_system.c

* Fix a memory leak in posix file handle implementation

* WT-2552 Use the correct flags when opening backup file.

* WT-2552 Add public API for pluggable filesystems

Simplify open-file error handling by calling the close function on the
handle, that way we won't forget to free all of the applicable memory
allocations.

* WT-2552 Add public API for pluggable filesystems

Simplify the directory-list method, don't pass in an include/exclude
file, if prefix is non-NULL, it implies we only want files matching
the prefix.

* WT-2552 Add public API for pluggable filesystems

Replace WT_FILE_HANDLE_POSIX.fallocate_{available,requires_locking} wiht
WT_FILE_HANDLE.fallocate and WT_FILE_HANDLE.fallocate_nolock.

Example code doesn't need to set WT_FILE_HANDLE methods to NULL, the
allocation does that.

Free the I/O buffer if open-handle allocation fails in the example code.

Remove snippets for WT_FILE_SYSTEM and WT_FILE_HANDLE methods, we're
not going to provide example code for them.

* WT-2552 Add public API for pluggable filesystems

Document we expect either ENOTSUP or EBUSY from optionally supported
APIs. Review/cleanups ENOTSUP/EBUSY returns from optionally supported
APIs.

Make WT_FILE_HANDLE.lock optional.

Don't configure or call the POSIX fadvise function on files configured
for direct I/O.

Rename __wt_filesize_name to __wt_size for consistency.

Update the spelling list.

* WT-2552 Add public API for pluggable filesystems

WT_FILE_HANDLE.truncate requires locking in all known implementations,
document it is not called concurrently with other operations.

* WT-2552 Add public API for pluggable filesystems

Don't terminate the filesystem unless we've actually configured one.

* WT-2552 Add public API for pluggable filesystems

Remove WT_FILE_SYSTEM and WT_FILE_HANDLE from SWIG so the test suite
can pass again.

* WT-2552 Add public API for pluggable filesystems

Merge __conn_load_early_extensions() and __conn_load_extensions().

Fix a problem where I moved the early extensions load to where it could
include the WiredTiger environment variable, but I didn't pass the built
cfg into the function.

* WT-2552 Add public API for pluggable filesystems

Linux build typo.

* WT-2552 Add public API for pluggable filesystems

Get rid of the "bool silent" argument to WT_FILE_SYSTEM.size by testing
for the file's existence before requesting the size (an extra system
call, but guaranteed to hit in the buffer cache at least).

* WT-2552 Add public API for pluggable filesystems

Naming consistency pass over the WT_FILE_SYSTEM functions.

* WT-2552 Add public API for pluggable filesystems

Fix a spin lock mismatch.

* WT-2552 Add public API for pluggable filesystems

Another spinlock mismatch.

* Update example pluggable file system.

Add a directory list implementation to the example, which uncovered
an issue with the API. The directory list API allocates memory that
is freed by WiredTiger, which I don't think is kosher.

* Change file-directory-sync to use reguar fsync.

The distinction in os_fs.i doesn't work with the filesystem API.

Also add directory_sync application to the example application.

* WT-2552 Add public API for pluggable filesystems

Whitespace.

* WT-2552 Add public API for pluggable filesystems

Rewrite __wt_free to not evaluate macro arguments multiple times.

* WT-2552 Add public API for pluggable filesystems

Simplify the directory-list functions: __wt_realloc_def() already
handles scaling the size of the allocations, there's no need to
involve a separate constant that increments the allocation size.

* WT-2552 Add public API for pluggable filesystems

Fix a grouping problem in a realloc call, we need to multiple the size
times the previously allocated slots + 10.

Fix buffer overrun, if "count" has already been incremented, the memset
would skip clearing the first slot and clear one slot past the end of
the buffer.

Remove a comment, realloc requires clearing allocated memory, it's not
paranoia.

* WT-2552 Add public API for pluggable filesystems

Add the mapping-cookie argument to the map-preload and map-discard
functions.

Change page-discard to stop reaching down through the block manager,
instead, provide a block-manager map-discard function that does the
work.

* WT-2552 Add public API for pluggable filesystems

Require a directory-list function.

Implement a directory-list function for the in-memory filesystem.

Consistency pass, make all the directory-list functions look the same.

* WT-2552 Add public API for pluggable filesystems

The WT_FILE_SYSTEM.{directory_sync, remove, rename} methods are not
required for read-only systems.

* WT-2552 Add public API for pluggable filesystems

Change the WT_FILE_SYSTEM.open_file file_type argument from a set of
constants to an enum.

This requires changing how we store connection direct I/O configuration
(the constants used to be flags stored in the WT_CONNECTION_IMPL), and
requiring all callers of __wt_open() do their own work to figure out if
WT_OPEN_DIRECTIO should be specified.

* WT-2552 Add public API for pluggable filesystems

Make no guarantees WT_FILE_SYSTEM and WT_FILE_HANDLE methods are
not called concurrently (except for WT_FILE_HANDLE::fallocate and
WT_FILE_HANDLE::fallocate_nolock).

Rewrite the in-memory FS code to lock across all methods (for example,
WT_FILE_HANDLE.close), that means including a reference to the enclosing
WT_FILE_SYSTEM in the WT_FILE_HANDLE structure so we can find a lock
without using the WT_CONNECTION_IMPL structure.

* WT-2552 Add public API for pluggable filesystems

Remove __wt_directory_sync_fh, it's no longer useful.

* WT-2552 Add public API for pluggable filesystems

Rename WT_INMEMORY_FILE_SYSTEM to WT_FILE_SYSTEM_INMEM, matching
WT_FILE_HANDLE_INMEM.

* WT-2552 Add public API for pluggable filesystems

Add WT_FILE_SYSTEM.directory_list_free, to free memory allocated
by WT_FILE_SYSTEM.direct_list.

Fix a memory leak in __log_archive_once (if __wt_readlock failed,
we leaked the directory-list memory).

* WT-2552 Add public API for pluggable filesystems

Typo, check WT_DIRECT_IO_LOG, not WT_DIRECT_IO_CHECKPOINT.

* WT-2552 Add public API for pluggable filesystems

Typo, unreachable code.

* WT-2552 Add public API for pluggable filesystems

We don't require WT_FILE_SYSTEM.{remove,rename} if the system is
read-only.

* Fix Windows build with pluggable file system.

Involved removing u_int from the public API.

* Fix line wrapping.

* Fix Windows terminate function.

* Forgot something in my last commit.

* Fix Windows munmap bug.

* Add new example to Windows build. Extend example to be more complete.

* Fix example loading on Windows

* Update documentation

* Add missing spell words

* Remove old comment.
2016-04-28 07:16:44 -04:00

195 lines
5.2 KiB
Python

# Output a C header file using the minimum number of distinct bits to ensure
# flags don't collide.
import os, re, sys
from dist import compare_srcfile
flags = {
###################################################
# Internal routine flag declarations
###################################################
'log_scan' : [
'LOGSCAN_FIRST',
'LOGSCAN_FROM_CKP',
'LOGSCAN_ONE',
'LOGSCAN_RECOVER',
],
'log_write' : [
'LOG_BACKGROUND',
'LOG_DSYNC',
'LOG_FLUSH',
'LOG_FSYNC',
'LOG_SYNC_ENABLED',
],
'page_read' : [
'READ_CACHE',
'READ_COMPACT',
'READ_NOTFOUND_OK',
'READ_NO_EMPTY',
'READ_NO_EVICT',
'READ_NO_GEN',
'READ_NO_WAIT',
'READ_PREV',
'READ_RESTART_OK',
'READ_SKIP_INTL',
'READ_SKIP_LEAF',
'READ_TRUNCATE',
'READ_WONT_NEED',
],
'rec_write' : [
'EVICT_IN_MEMORY',
'EVICT_LOOKASIDE',
'EVICT_UPDATE_RESTORE',
'EVICTING',
'VISIBILITY_ERR',
],
'txn_log_checkpoint' : [
'TXN_LOG_CKPT_CLEANUP',
'TXN_LOG_CKPT_PREPARE',
'TXN_LOG_CKPT_START',
'TXN_LOG_CKPT_STOP',
'TXN_LOG_CKPT_SYNC',
],
'verbose' : [
'VERB_API',
'VERB_BLOCK',
'VERB_CHECKPOINT',
'VERB_COMPACT',
'VERB_EVICT',
'VERB_EVICTSERVER',
'VERB_FILEOPS',
'VERB_HANDLEOPS',
'VERB_LOG',
'VERB_LSM',
'VERB_LSM_MANAGER',
'VERB_METADATA',
'VERB_MUTEX',
'VERB_OVERFLOW',
'VERB_READ',
'VERB_REBALANCE',
'VERB_RECONCILE',
'VERB_RECOVERY',
'VERB_SALVAGE',
'VERB_SHARED_CACHE',
'VERB_SPLIT',
'VERB_TEMPORARY',
'VERB_TRANSACTION',
'VERB_VERIFY',
'VERB_VERSION',
'VERB_WRITE',
],
###################################################
# Structure flag declarations
###################################################
'conn' : [
'CONN_CACHE_POOL',
'CONN_CKPT_SYNC',
'CONN_CLOSING',
'CONN_EVICTION_RUN',
'CONN_IN_MEMORY',
'CONN_LAS_OPEN',
'CONN_LEAK_MEMORY',
'CONN_LOG_SERVER_RUN',
'CONN_LSM_MERGE',
'CONN_PANIC',
'CONN_READONLY',
'CONN_SERVER_ASYNC',
'CONN_SERVER_CHECKPOINT',
'CONN_SERVER_LSM',
'CONN_SERVER_RUN',
'CONN_SERVER_STATISTICS',
'CONN_SERVER_SWEEP',
'CONN_WAS_BACKUP',
],
'session' : [
'SESSION_CAN_WAIT',
'SESSION_CLEAR_EVICT_WALK',
'SESSION_INTERNAL',
'SESSION_LOCK_NO_WAIT',
'SESSION_LOCKED_CHECKPOINT',
'SESSION_LOCKED_HANDLE_LIST',
'SESSION_LOCKED_METADATA',
'SESSION_LOCKED_SCHEMA',
'SESSION_LOCKED_SLOT',
'SESSION_LOCKED_TABLE',
'SESSION_LOCKED_TURTLE',
'SESSION_LOGGING_INMEM',
'SESSION_LOOKASIDE_CURSOR',
'SESSION_NO_CACHE',
'SESSION_NO_DATA_HANDLES',
'SESSION_NO_EVICTION',
'SESSION_NO_LOGGING',
'SESSION_NO_SCHEMA_LOCK',
'SESSION_QUIET_CORRUPT_FILE',
'SESSION_SERVER_ASYNC',
],
}
flag_cnt = {} # Dictionary [flag] : [reference count]
flag_name = {} # Dictionary [flag] : [name ...]
name_mask = {} # Dictionary [name] : [used flag mask]
# Step through the flags dictionary and build our local dictionaries.
for method in flags.items():
name_mask[method[0]] = 0x0
for flag in method[1]:
if flag == '__NONE__':
continue
if flag not in flag_cnt:
flag_cnt[flag] = 1
flag_name[flag] = []
else:
flag_cnt[flag] += 1
flag_name[flag].append(method[0])
# Create list of possible bit masks.
bits = [2 ** i for i in range(0, 32)]
# Walk the list of flags in reverse, sorted-by-reference count order. For
# each flag, find a bit that's not currently in use by any method using the
# flag.
flag_bit = {} # Dictionary [flag] : [bit value]
for f in sorted(flag_cnt.items(), key = lambda k_v : (-k_v[1], k_v[0])):
mask = 0xffffffff
for m in flag_name[f[0]]:
mask &= ~name_mask[m]
if mask == 0:
print >>sys.stderr,\
"flags.py: ran out of flags at " + m + " method",
sys.exit(1)
for b in bits:
if mask & b:
mask = b
break
flag_bit[f[0]] = mask
for m in flag_name[f[0]]:
name_mask[m] |= mask
# Print out the flag masks in hex.
# Assumes tab stops set to 8 characters.
flag_info = ''
for f in sorted(flag_cnt.items()):
flag_info += "#define\tWT_%s%s%#010x\n" %\
(f[0],\
"\t" * max(1, 6 - int((len('WT_') + len(f[0])) / 8)),\
flag_bit[f[0]])
# Update the wiredtiger.in file with the flags information.
tmp_file = '__tmp'
tfile = open(tmp_file, 'w')
skip = 0
for line in open('../src/include/flags.h', 'r'):
if skip:
if line.count('flags section: END'):
tfile.write('/*\n' + line)
skip = 0
else:
tfile.write(line)
if line.count('flags section: BEGIN'):
skip = 1
tfile.write(' */\n')
tfile.write(flag_info)
tfile.close()
compare_srcfile(tmp_file, '../src/include/flags.h')