* WT-2552 Add public API for pluggable filesystems
Not yet compiling. The main parts of this change should be here,
but it involved extensive parameter re-organization. There are also
a number of layering violations between our existing file system
implementations and the WT_FH, that aren't possible with the new
structure.
There are a number of specific todo comments in the code. One of the main
issues is that the in-memory file system had a special close semantic
that relied on WiredTiger handle tracking. The in-memory file-system should
do it's own tracking of file handles, I've gone part way down that road by
adding a queue for closed handles. Need to also add in live handles, and
manage the queue as appropriate.
I haven't created an example application that uses the new API yet.
* WT-2552 Add public API for pluggable filesystems
I always forget you have to remove the already-built html files when
changing PREDEFINED, add a reminder to the complaint.
* WT-2552 Add public API for pluggable filesystems
You have to remove the .js files, too.
* WT-2552 Add public API for pluggable filesystems
Make dist/s_all run cleanly.
* WT-2552 Add public API for pluggable filesystems
Whitespace.
* WT-2552 Add public API for pluggable filesystems
Make it compile/build/lint.
* WT-2552 Add public API for pluggable filesystems
block_write.c: In function '__wt_block_extend':
block_write.c:130:71: error: missing terminating ' character [-Werror]
* WT-2552 Add public API for pluggable filesystems
os_fs_inmemory.c: In function '__im_file_truncate':
os_fs_inmemory.c:344:10: error: 'session' is used uninitialized in this
function [-Werror=uninitialized]
* WT-2552 Add public API for pluggable filesystems
os_fs.c: In function '__posix_directory_sync':
os_fs.c:92:10: error: 'session' is used uninitialized in this function
[-Werror=uninitialized]
* WT-2552 Add public API for pluggable filesystems
Go back to using bool types in the file-system API, this requires we add
<stdbool.h> to the "standard" wiredtiger.h includes.
Consistently use wt_session to represent a WT_SESSION, we were using
"wtsession" in some places.
Make a pass over the Windows code, but I'm sure it doesn't compile yet.
* WT-2552 Add public API for pluggable filesystems
Fix up another couple of bool types.
* WT-2552 Add public API for pluggable filesystems
Move the file naming work out of the underlying filesystem functions,
the calls to __wt_filename are now in the upper-level code,n os_fs.i;
that means the filesystem code is no longer responsible for figuring out
paths. This is cleaner, although the directory-sync call is a bit of a
kluge, and I've commimtted us to handling NULL filesystem methods.
With this set of changes, in-memory runs again.
More Windows naming fixes.
* WT-2552 Add public API for pluggable filesystems
os_fs.c: In function '__posix_directory_sync':
os_fs.c:96:3: error: label 'err' used but not defined
* WT-2552 Add public API for pluggable filesystems
Pull out another call to __wt_filename() from the filesystem-dependent
code.
* WT-2552 Add public API for pluggable filesystems
Consistently check for missing file-system methods when doing
file-system calls.
Other minor lint & cleanup.
* WT-2552 Add public API for pluggable filesystems
Change the in-memory code to maintain a complete list of the files it
has ever opened, and depend on that list instead of reaching up into the
common layer for the WT_FH handle list.
This means __wt_handle_search is only used by the common WT_FH handle
code, simplify it, and add a __wt_handle_is_open function that can be
called for diagnostic purposes (to check for open files that are being
renamed or removed, for example).
* Fix comiler warning and ignore the file system API in Java
* Flesh out the example file system implementation.
* Add in some plumbing for set_file_system in wiredtiger_open.
* WT-2552 Add public API for pluggable filesystems
Whitespace.
* WT-2552 Add public API for pluggable filesystems
WT_CONFIG_ITEM.val isn't a boolean, don't use boolean types in
equal/not-equal comparisons.
* WT-2552 Add public API for pluggable filesystems
Remove unused #includes.
Increment/decrement the DEMO_FILE_SYSTEM.{opened,closed}_file_count.
Allocate demo structures, they're larger than the underlying structures.
Swap the number/size calloc arguments, number comes first.
Fix a couple of statics.
* WT-2552 Add public API for pluggable filesystems
Use %u instead of casting to %d.
* WT-2552 Add public API for pluggable filesystems
Add ex_file_system.c to the list of example programs.
* WT-2552 Add public API for pluggable filesystems
Change ex_file_system.c to not require <wt_internal.h>: strip down a
copy of FreeBSD's <queue.h> for local inclusion, rewrite a few other
minor pieces of code.
* WT-2552 Add public API for pluggable filesystems
Update spell check info
* WT-2552 Add public API for pluggable filesystems
__conn_load_extensions() shouldn't set the "early" boolean to true.
* WT-2552 Add public API for pluggable filesystems
Don't indirect through a NULL pointer if "local" was set and no path was
specified, always set the name to something useful.
* WT-2552 Add public API for pluggable filesystems
Don't indirect through a NULL pointer if "local" was set and no path was
specified, always set the name to something useful.
* WT-2552 Add public API for pluggable filesystems
wt_off_t vs. size_t conversion lint.
* WT-2552 Add public API for pluggable filesystems
Add -rdynamic to the load for ex_file_system, the main executable
symbols are not exported by default.
* WT-2552 Add public API for pluggable filesystems
The underlying handle name includes the enclosing directory,
compare against the WT_FH.name field instead.
* WT-2552 Add public API for pluggable filesystems
demo_fs_rename should return 0 if successful, simplify error handling
Don't bother casting arguments to free(), it's not necessary.
* WT-2552 Add public API for pluggable filesystems
General WT_FILE_SYSTEM cleanup.
Move OS initialization into the wiredtiger_open() code (the
os_common/os_init.c file is no longer needed).
Allow early-load extensions to be part of the environment settings,
matching the "in-memory" and "readonly" configurations.
Syntax check the set of a file-system, remove tests for NULL methods in
the file-system structure unless it's legal for them to be NULL.
Windows, POSIX and in-memory file systems now set WT_FILE_SYSTEM.terminate,
call that function to cleanup when discarding a WT_CONNECTION.
Export file-type and open-flags constants for WT_FILE_SYSTEM.open_file,
sort the WT_FILE_SYSTEM methods, do an editing pass.
Change the WT_FILE_HANDLE type from (const char *) to (char *), it's
"owned" by the underlying layer, and it's simpler that way.
Minor (untested) cleanup of the Windows WT_FILE_SYSTEM.open-file method.
* WT-2552 Add public API for pluggable filesystems
Export the advise argument #defines for the WT_FILE_HANDLE.fadvise method.
Sort the WT_FILE_HANDLE methods.
* WT-2552 Add public API for pluggable filesystems
Clean up and simplify WT_FILE_SYSTEM/WT_FILE_HANDLE documentation's
description of the handles.
* WT-2552 Add public API for pluggable filesystems
WT_FILE_HANDLE.close is a required function (at the least, it
has to free the memory).
WT_FILE_HANDLE.fadvise isn't a required function, if it's not
configured, don't call it.
* WT-2552 Add public API for pluggable filesystems
The WT_FILE_HANDLE.lock function is required.
Change the __wt_open() signature to match WT_FILE_SYSTEM.open_file().
* WT-2552 Add public API for pluggable filesystems
Rework all of the WT_FILE_HANDLE mapped region methods to be optional.
* WT-2552 Add public API for pluggable filesystems
The WT_FILE_HANDLE.{read,size} methods are required.
The WT_FILE_HANDLE.sync method is not required.
Split the WT_FILE_HANDLE.sync method into .sync and .sync_nowait versions,
it makes the upper-level code simpler (Windows supports .sync but doesn't
support .sync_nowait).
* WT-2552 Add public API for pluggable filesystems
The WT_FILE_HANDLE.{truncate,write} methods are required IFF the file
is not readonly.
* WT-2552 Add public API for pluggable filesystems
POSIX shouldn't declare a no-sync handle function unless the
sync_file_range system call is available.
* WT-2552 Add public API for pluggable filesystems
Typo, missing semi-colon.
* Fix a bug in ex_file_system.c
* Fix a memory leak in posix file handle implementation
* WT-2552 Use the correct flags when opening backup file.
* WT-2552 Add public API for pluggable filesystems
Simplify open-file error handling by calling the close function on the
handle, that way we won't forget to free all of the applicable memory
allocations.
* WT-2552 Add public API for pluggable filesystems
Simplify the directory-list method, don't pass in an include/exclude
file, if prefix is non-NULL, it implies we only want files matching
the prefix.
* WT-2552 Add public API for pluggable filesystems
Replace WT_FILE_HANDLE_POSIX.fallocate_{available,requires_locking} wiht
WT_FILE_HANDLE.fallocate and WT_FILE_HANDLE.fallocate_nolock.
Example code doesn't need to set WT_FILE_HANDLE methods to NULL, the
allocation does that.
Free the I/O buffer if open-handle allocation fails in the example code.
Remove snippets for WT_FILE_SYSTEM and WT_FILE_HANDLE methods, we're
not going to provide example code for them.
* WT-2552 Add public API for pluggable filesystems
Document we expect either ENOTSUP or EBUSY from optionally supported
APIs. Review/cleanups ENOTSUP/EBUSY returns from optionally supported
APIs.
Make WT_FILE_HANDLE.lock optional.
Don't configure or call the POSIX fadvise function on files configured
for direct I/O.
Rename __wt_filesize_name to __wt_size for consistency.
Update the spelling list.
* WT-2552 Add public API for pluggable filesystems
WT_FILE_HANDLE.truncate requires locking in all known implementations,
document it is not called concurrently with other operations.
* WT-2552 Add public API for pluggable filesystems
Don't terminate the filesystem unless we've actually configured one.
* WT-2552 Add public API for pluggable filesystems
Remove WT_FILE_SYSTEM and WT_FILE_HANDLE from SWIG so the test suite
can pass again.
* WT-2552 Add public API for pluggable filesystems
Merge __conn_load_early_extensions() and __conn_load_extensions().
Fix a problem where I moved the early extensions load to where it could
include the WiredTiger environment variable, but I didn't pass the built
cfg into the function.
* WT-2552 Add public API for pluggable filesystems
Linux build typo.
* WT-2552 Add public API for pluggable filesystems
Get rid of the "bool silent" argument to WT_FILE_SYSTEM.size by testing
for the file's existence before requesting the size (an extra system
call, but guaranteed to hit in the buffer cache at least).
* WT-2552 Add public API for pluggable filesystems
Naming consistency pass over the WT_FILE_SYSTEM functions.
* WT-2552 Add public API for pluggable filesystems
Fix a spin lock mismatch.
* WT-2552 Add public API for pluggable filesystems
Another spinlock mismatch.
* Update example pluggable file system.
Add a directory list implementation to the example, which uncovered
an issue with the API. The directory list API allocates memory that
is freed by WiredTiger, which I don't think is kosher.
* Change file-directory-sync to use reguar fsync.
The distinction in os_fs.i doesn't work with the filesystem API.
Also add directory_sync application to the example application.
* WT-2552 Add public API for pluggable filesystems
Whitespace.
* WT-2552 Add public API for pluggable filesystems
Rewrite __wt_free to not evaluate macro arguments multiple times.
* WT-2552 Add public API for pluggable filesystems
Simplify the directory-list functions: __wt_realloc_def() already
handles scaling the size of the allocations, there's no need to
involve a separate constant that increments the allocation size.
* WT-2552 Add public API for pluggable filesystems
Fix a grouping problem in a realloc call, we need to multiple the size
times the previously allocated slots + 10.
Fix buffer overrun, if "count" has already been incremented, the memset
would skip clearing the first slot and clear one slot past the end of
the buffer.
Remove a comment, realloc requires clearing allocated memory, it's not
paranoia.
* WT-2552 Add public API for pluggable filesystems
Add the mapping-cookie argument to the map-preload and map-discard
functions.
Change page-discard to stop reaching down through the block manager,
instead, provide a block-manager map-discard function that does the
work.
* WT-2552 Add public API for pluggable filesystems
Require a directory-list function.
Implement a directory-list function for the in-memory filesystem.
Consistency pass, make all the directory-list functions look the same.
* WT-2552 Add public API for pluggable filesystems
The WT_FILE_SYSTEM.{directory_sync, remove, rename} methods are not
required for read-only systems.
* WT-2552 Add public API for pluggable filesystems
Change the WT_FILE_SYSTEM.open_file file_type argument from a set of
constants to an enum.
This requires changing how we store connection direct I/O configuration
(the constants used to be flags stored in the WT_CONNECTION_IMPL), and
requiring all callers of __wt_open() do their own work to figure out if
WT_OPEN_DIRECTIO should be specified.
* WT-2552 Add public API for pluggable filesystems
Make no guarantees WT_FILE_SYSTEM and WT_FILE_HANDLE methods are
not called concurrently (except for WT_FILE_HANDLE::fallocate and
WT_FILE_HANDLE::fallocate_nolock).
Rewrite the in-memory FS code to lock across all methods (for example,
WT_FILE_HANDLE.close), that means including a reference to the enclosing
WT_FILE_SYSTEM in the WT_FILE_HANDLE structure so we can find a lock
without using the WT_CONNECTION_IMPL structure.
* WT-2552 Add public API for pluggable filesystems
Remove __wt_directory_sync_fh, it's no longer useful.
* WT-2552 Add public API for pluggable filesystems
Rename WT_INMEMORY_FILE_SYSTEM to WT_FILE_SYSTEM_INMEM, matching
WT_FILE_HANDLE_INMEM.
* WT-2552 Add public API for pluggable filesystems
Add WT_FILE_SYSTEM.directory_list_free, to free memory allocated
by WT_FILE_SYSTEM.direct_list.
Fix a memory leak in __log_archive_once (if __wt_readlock failed,
we leaked the directory-list memory).
* WT-2552 Add public API for pluggable filesystems
Typo, check WT_DIRECT_IO_LOG, not WT_DIRECT_IO_CHECKPOINT.
* WT-2552 Add public API for pluggable filesystems
Typo, unreachable code.
* WT-2552 Add public API for pluggable filesystems
We don't require WT_FILE_SYSTEM.{remove,rename} if the system is
read-only.
* Fix Windows build with pluggable file system.
Involved removing u_int from the public API.
* Fix line wrapping.
* Fix Windows terminate function.
* Forgot something in my last commit.
* Fix Windows munmap bug.
* Add new example to Windows build. Extend example to be more complete.
* Fix example loading on Windows
* Update documentation
* Add missing spell words
* Remove old comment.
195 lines
5.2 KiB
Python
195 lines
5.2 KiB
Python
# Output a C header file using the minimum number of distinct bits to ensure
|
|
# flags don't collide.
|
|
|
|
import os, re, sys
|
|
from dist import compare_srcfile
|
|
|
|
flags = {
|
|
###################################################
|
|
# Internal routine flag declarations
|
|
###################################################
|
|
'log_scan' : [
|
|
'LOGSCAN_FIRST',
|
|
'LOGSCAN_FROM_CKP',
|
|
'LOGSCAN_ONE',
|
|
'LOGSCAN_RECOVER',
|
|
],
|
|
'log_write' : [
|
|
'LOG_BACKGROUND',
|
|
'LOG_DSYNC',
|
|
'LOG_FLUSH',
|
|
'LOG_FSYNC',
|
|
'LOG_SYNC_ENABLED',
|
|
],
|
|
'page_read' : [
|
|
'READ_CACHE',
|
|
'READ_COMPACT',
|
|
'READ_NOTFOUND_OK',
|
|
'READ_NO_EMPTY',
|
|
'READ_NO_EVICT',
|
|
'READ_NO_GEN',
|
|
'READ_NO_WAIT',
|
|
'READ_PREV',
|
|
'READ_RESTART_OK',
|
|
'READ_SKIP_INTL',
|
|
'READ_SKIP_LEAF',
|
|
'READ_TRUNCATE',
|
|
'READ_WONT_NEED',
|
|
],
|
|
'rec_write' : [
|
|
'EVICT_IN_MEMORY',
|
|
'EVICT_LOOKASIDE',
|
|
'EVICT_UPDATE_RESTORE',
|
|
'EVICTING',
|
|
'VISIBILITY_ERR',
|
|
],
|
|
'txn_log_checkpoint' : [
|
|
'TXN_LOG_CKPT_CLEANUP',
|
|
'TXN_LOG_CKPT_PREPARE',
|
|
'TXN_LOG_CKPT_START',
|
|
'TXN_LOG_CKPT_STOP',
|
|
'TXN_LOG_CKPT_SYNC',
|
|
],
|
|
'verbose' : [
|
|
'VERB_API',
|
|
'VERB_BLOCK',
|
|
'VERB_CHECKPOINT',
|
|
'VERB_COMPACT',
|
|
'VERB_EVICT',
|
|
'VERB_EVICTSERVER',
|
|
'VERB_FILEOPS',
|
|
'VERB_HANDLEOPS',
|
|
'VERB_LOG',
|
|
'VERB_LSM',
|
|
'VERB_LSM_MANAGER',
|
|
'VERB_METADATA',
|
|
'VERB_MUTEX',
|
|
'VERB_OVERFLOW',
|
|
'VERB_READ',
|
|
'VERB_REBALANCE',
|
|
'VERB_RECONCILE',
|
|
'VERB_RECOVERY',
|
|
'VERB_SALVAGE',
|
|
'VERB_SHARED_CACHE',
|
|
'VERB_SPLIT',
|
|
'VERB_TEMPORARY',
|
|
'VERB_TRANSACTION',
|
|
'VERB_VERIFY',
|
|
'VERB_VERSION',
|
|
'VERB_WRITE',
|
|
],
|
|
|
|
###################################################
|
|
# Structure flag declarations
|
|
###################################################
|
|
'conn' : [
|
|
'CONN_CACHE_POOL',
|
|
'CONN_CKPT_SYNC',
|
|
'CONN_CLOSING',
|
|
'CONN_EVICTION_RUN',
|
|
'CONN_IN_MEMORY',
|
|
'CONN_LAS_OPEN',
|
|
'CONN_LEAK_MEMORY',
|
|
'CONN_LOG_SERVER_RUN',
|
|
'CONN_LSM_MERGE',
|
|
'CONN_PANIC',
|
|
'CONN_READONLY',
|
|
'CONN_SERVER_ASYNC',
|
|
'CONN_SERVER_CHECKPOINT',
|
|
'CONN_SERVER_LSM',
|
|
'CONN_SERVER_RUN',
|
|
'CONN_SERVER_STATISTICS',
|
|
'CONN_SERVER_SWEEP',
|
|
'CONN_WAS_BACKUP',
|
|
],
|
|
'session' : [
|
|
'SESSION_CAN_WAIT',
|
|
'SESSION_CLEAR_EVICT_WALK',
|
|
'SESSION_INTERNAL',
|
|
'SESSION_LOCK_NO_WAIT',
|
|
'SESSION_LOCKED_CHECKPOINT',
|
|
'SESSION_LOCKED_HANDLE_LIST',
|
|
'SESSION_LOCKED_METADATA',
|
|
'SESSION_LOCKED_SCHEMA',
|
|
'SESSION_LOCKED_SLOT',
|
|
'SESSION_LOCKED_TABLE',
|
|
'SESSION_LOCKED_TURTLE',
|
|
'SESSION_LOGGING_INMEM',
|
|
'SESSION_LOOKASIDE_CURSOR',
|
|
'SESSION_NO_CACHE',
|
|
'SESSION_NO_DATA_HANDLES',
|
|
'SESSION_NO_EVICTION',
|
|
'SESSION_NO_LOGGING',
|
|
'SESSION_NO_SCHEMA_LOCK',
|
|
'SESSION_QUIET_CORRUPT_FILE',
|
|
'SESSION_SERVER_ASYNC',
|
|
],
|
|
}
|
|
|
|
flag_cnt = {} # Dictionary [flag] : [reference count]
|
|
flag_name = {} # Dictionary [flag] : [name ...]
|
|
name_mask = {} # Dictionary [name] : [used flag mask]
|
|
|
|
# Step through the flags dictionary and build our local dictionaries.
|
|
for method in flags.items():
|
|
name_mask[method[0]] = 0x0
|
|
for flag in method[1]:
|
|
if flag == '__NONE__':
|
|
continue
|
|
if flag not in flag_cnt:
|
|
flag_cnt[flag] = 1
|
|
flag_name[flag] = []
|
|
else:
|
|
flag_cnt[flag] += 1
|
|
flag_name[flag].append(method[0])
|
|
|
|
# Create list of possible bit masks.
|
|
bits = [2 ** i for i in range(0, 32)]
|
|
|
|
# Walk the list of flags in reverse, sorted-by-reference count order. For
|
|
# each flag, find a bit that's not currently in use by any method using the
|
|
# flag.
|
|
flag_bit = {} # Dictionary [flag] : [bit value]
|
|
for f in sorted(flag_cnt.items(), key = lambda k_v : (-k_v[1], k_v[0])):
|
|
mask = 0xffffffff
|
|
for m in flag_name[f[0]]:
|
|
mask &= ~name_mask[m]
|
|
if mask == 0:
|
|
print >>sys.stderr,\
|
|
"flags.py: ran out of flags at " + m + " method",
|
|
sys.exit(1)
|
|
for b in bits:
|
|
if mask & b:
|
|
mask = b
|
|
break
|
|
flag_bit[f[0]] = mask
|
|
for m in flag_name[f[0]]:
|
|
name_mask[m] |= mask
|
|
|
|
# Print out the flag masks in hex.
|
|
# Assumes tab stops set to 8 characters.
|
|
flag_info = ''
|
|
for f in sorted(flag_cnt.items()):
|
|
flag_info += "#define\tWT_%s%s%#010x\n" %\
|
|
(f[0],\
|
|
"\t" * max(1, 6 - int((len('WT_') + len(f[0])) / 8)),\
|
|
flag_bit[f[0]])
|
|
|
|
# Update the wiredtiger.in file with the flags information.
|
|
tmp_file = '__tmp'
|
|
tfile = open(tmp_file, 'w')
|
|
skip = 0
|
|
for line in open('../src/include/flags.h', 'r'):
|
|
if skip:
|
|
if line.count('flags section: END'):
|
|
tfile.write('/*\n' + line)
|
|
skip = 0
|
|
else:
|
|
tfile.write(line)
|
|
if line.count('flags section: BEGIN'):
|
|
skip = 1
|
|
tfile.write(' */\n')
|
|
tfile.write(flag_info)
|
|
tfile.close()
|
|
compare_srcfile(tmp_file, '../src/include/flags.h')
|