Commit Graph

437 Commits

Author SHA1 Message Date
ivan
63d5315e1c
Merge branch 'master' into master 2019-05-23 14:02:04 +08:00
Chris Mumford
c00e177f36 Guard DBImpl::versions_ by mutex_.
mutex_ was already acquired before accessing DBImpl::versions_ in all
but one place: DBImpl::GetApproximateSizes. This change requires mutex_
to be held before accessing versions_.

PiperOrigin-RevId: 248390814
2019-05-16 12:07:21 -07:00
Chris Mumford
1d0b101165 Converted two for-loops to while-loops.
Converted `for (;<condition>;)` to `while (<condition>)`.

PiperOrigin-RevId: 247950510
2019-05-13 13:51:11 -07:00
Chris Mumford
28e6d238be Switch to using C++ 11 override specifier.
PiperOrigin-RevId: 247491163
2019-05-09 14:11:06 -07:00
Chris Mumford
85cd40d108 Added unit test for InternalKey::DecodeFrom with empty string.
PiperOrigin-RevId: 247483339
2019-05-09 14:10:55 -07:00
Chris Mumford
1aae5c9f29 Merge pull request #411 from proller:assert1
PiperOrigin-RevId: 247424040
2019-05-09 08:37:49 -07:00
Chris Mumford
b7b86baec9 Using std::ostringstream in key DebugString.
Switching from snprintf to std::ostringstream eliminates
cast warning for (unsigned long long).

PiperOrigin-RevId: 247326681
2019-05-08 17:36:35 -07:00
Chris Mumford
3e6c000e18 Merge pull request #457 from jellor:patch-2
PiperOrigin-RevId: 247261470
2019-05-08 17:36:22 -07:00
果冻
1d94fe2f4d
Merge branch 'master' into patch-2 2019-05-09 00:08:49 +08:00
Victor Costan
27dc99fb26 Fix EnvPosix tests on Travis CI.
The previous attempt of having EnvPosix use O_CLOEXEC (close-on-exec()) when opening file descriptors added tests that relied on procfs, which is Linux-specific. These tests failed on macOS. Unfortunately, the test failures were not caught due to a (since fixed) error in our Travis CI configuration.

This CL re-structures the tests to only rely on POSIX features. Since there is no POSIX-compliant way to get a file name/path out of a file descriptor, this CL breaks up the O_CLOEXEC test into multiple tests, where each Env method that creates an FD gets its own test. This is intended to make it easier to find and fix errors in Env implementations.

This CL also fixes the implementation of NewLogger() to use O_CLOEXEC on macOS. The current implementation passes "we" to fopen(), but the macOS standard C library does not implement the "e" flag yet.

PiperOrigin-RevId: 247088953
2019-05-07 14:20:31 -07:00
Chris Mumford
9521545b06 Formatting changes for prior O_CLOEXEC fix.
Two minor corrections to correct the 900f7d37eb commit
to conform to the Google C++ style guide.

PiperOrigin-RevId: 246907647
2019-05-06 15:34:20 -07:00
Chris Mumford
900f7d37eb Merge pull request #624 from adam-azarchs:master
PiperOrigin-RevId: 246903086
2019-05-06 15:00:48 -07:00
Victor Costan
a7528a5d2b Clean up util/coding.{h,cc}.
1) Inline EncodeFixed{32,64}(). They emit single machine instructions on 64-bit processors.
2) Remove size narrowing compiler warnings from DecodeFixed{32,64}().
3) Add comments explaining the current state of optimizations in compilers we care about.
4) Change C-style includes, like <stdint.h>, to C++ style, like <cstdint>.
5) memcpy -> std::memcpy.

The optimization comments are based on https://godbolt.org/z/RdIqS1. The missed optimization opportunities in clang have been reported as https://bugs.llvm.org/show_bug.cgi?id=41761

The change does not have significant impact on benchmarks. Results below.

LevelDB:    version 1.22
Date:       Mon May  6 10:42:18 2019
CPU:        72 * Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz
CPUCache:   25344 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    1000000
RawSize:    110.6 MB (estimated)
FileSize:   62.9 MB (estimated)

With change
------------------------------------------------
fillseq      :       2.327 micros/op;   47.5 MB/s
fillsync     :    4185.526 micros/op;    0.0 MB/s (1000 ops)
fillrandom   :       3.662 micros/op;   30.2 MB/s
overwrite    :       4.261 micros/op;   26.0 MB/s
readrandom   :       4.239 micros/op; (1000000 of 1000000 found)
readrandom   :       3.649 micros/op; (1000000 of 1000000 found)
readseq      :       0.174 micros/op;  636.7 MB/s
readreverse  :       0.271 micros/op;  408.7 MB/s
compact      :  570495.000 micros/op;
readrandom   :       2.735 micros/op; (1000000 of 1000000 found)
readseq      :       0.118 micros/op;  937.3 MB/s
readreverse  :       0.190 micros/op;  583.7 MB/s
fill100K     :     860.164 micros/op;  110.9 MB/s (1000 ops)
crc32c       :       1.131 micros/op; 3455.2 MB/s (4K per op)
snappycomp   :       3.034 micros/op; 1287.5 MB/s (output: 55.1%)
snappyuncomp :       0.544 micros/op; 7176.0 MB/s

Baseline
------------------------------------------------
fillseq      :       2.365 micros/op;   46.8 MB/s
fillsync     :    4240.165 micros/op;    0.0 MB/s (1000 ops)
fillrandom   :       3.244 micros/op;   34.1 MB/s
overwrite    :       4.153 micros/op;   26.6 MB/s
readrandom   :       4.698 micros/op; (1000000 of 1000000 found)
readrandom   :       4.065 micros/op; (1000000 of 1000000 found)
readseq      :       0.192 micros/op;  576.3 MB/s
readreverse  :       0.286 micros/op;  386.7 MB/s
compact      :  635979.000 micros/op;
readrandom   :       3.264 micros/op; (1000000 of 1000000 found)
readseq      :       0.169 micros/op;  652.8 MB/s
readreverse  :       0.213 micros/op;  519.5 MB/s
fill100K     :    1055.367 micros/op;   90.4 MB/s (1000 ops)
crc32c       :       1.353 micros/op; 2887.3 MB/s (4K per op)
snappycomp   :       3.036 micros/op; 1286.7 MB/s (output: 55.1%)
snappyuncomp :       0.540 micros/op; 7238.6 MB/s
PiperOrigin-RevId: 246856811
2019-05-06 11:23:02 -07:00
Chris Mumford
142035edd4 Initialize Stats::start_ before first use in Stats::Start().
Avoids a use before initialization error. This fixes issue #676.

PiperOrigin-RevId: 246855204
2019-05-06 10:52:16 -07:00
Victor Costan
e22b1cec6e Merge pull request #365 from allangj:c-strict-prototypes
PiperOrigin-RevId: 246848402
2019-05-06 10:29:40 -07:00
allangj
cd1ec032cd Add argument definition for void c functions.
Allow the use c.h on projects with -Wstrict-prototypes
Modify CMakelist to include -Wstrict-prototypes
2019-05-05 18:13:39 -06:00
Victor Costan
4bd052d7e8 Consolidate benchmark code to benchmarks/.
Currently, the benchmark used to assess leveldb changes lives in db/. The codebase also contains two benchmarks against other database engines in doc/bench/. Moving all the benchmarks in one place opens up the way for extracting common code.

PiperOrigin-RevId: 246737541
2019-05-05 12:59:23 -07:00
Victor Costan
506b1722ef Convert missed virtual -> override in db_test.cc.
PiperOrigin-RevId: 246680419
2019-05-04 18:05:53 -07:00
Victor Costan
24424a1ef2 Style cleanup.
1) Convert iterator-based for loops to C++11 foreach loops.
2) Convert "void operator=" to "T& operator=".
3) Switch from copy operators from private to public deleted.
4) Switch from empty ctors / dtors to "= default" where appropriate.

PiperOrigin-RevId: 246679195
2019-05-04 17:42:20 -07:00
Victor Costan
9a56c49ed4 Merge pull request #679 from smartxworks:optimize-readseq
PiperOrigin-RevId: 246668103
2019-05-04 16:53:17 -07:00
Victor Costan
abf441b657 Merge pull request #278 from wankai:master
PiperOrigin-RevId: 246591372
2019-05-04 02:12:27 -07:00
Chris Mumford
78b39d68c1 Bump the version number from 1.21 to 1.22.
PiperOrigin-RevId: 246558281
2019-05-03 13:24:26 -07:00
Chris Mumford
9bd23c7676 Correct class/structure declaration order.
1. Correct the class/struct declaration order to be IAW
   the Google C++ style guide[1].
2. For non-copyable classes, switched from non-implemented
   private methods to explicitly deleted[2] methods.
3. Minor const and member initialization fixes.

[1] https://google.github.io/styleguide/cppguide.html#Declaration_Order
[2] http://eel.is/c++draft/dcl.fct.def.delete

PiperOrigin-RevId: 246521844
2019-05-03 09:48:57 -07:00
Chris Mumford
c784d63b93 Moved port/README to port/README.md.
Easier to read on sites supporting Markdown (i.e. GitHub).

PiperOrigin-RevId: 246385089
2019-05-02 19:05:04 -07:00
Chris Mumford
297e66afc1 Format all files IAW the Google C++ Style Guide.
Use clang-format to correct formatting to be in agreement with the [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html). Doing this simplifies the process of accepting changes. Also fixed a few warnings flagged by clang-tidy.

PiperOrigin-RevId: 246350737
2019-05-02 19:04:50 -07:00
Victor Costan
3724030179 Update Travis CI configuration.
The Travis configuration:
1) Installs recent versions of clang and GCC.
2) Sets up the environment so that CMake picks up the installed
   compilers. Previously, the pre-installed clang compiler was used
   instead.
3) Requests a modern macOS image that has all the headers needed by GCC.

The CL also removes now-unnecessary old workarounds from the
Travis configuration.

PiperOrigin-RevId: 245831188
2019-04-29 15:38:15 -07:00
Kyle Zhang
d3d1c8a0f4 don't check current key in DBIter::Next()
When iter_ is pointing to current key, we can safely move to the next
key to avoid checking current key, which is of course not necessary.

Benchmark shows that 'readseq' has about 8% performance improvement.

Without patch:

>./db_bench --benchmarks=readseq --num=$((4<<20)) --db=/tmp/db --use_existing_db=1
LevelDB:    version 1.21
Date:       Thu Apr 25 09:37:21 2019
CPU:        32 * Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
CPUCache:   20480 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    4194304
RawSize:    464.0 MB (estimated)
FileSize:   264.0 MB (estimated)
------------------------------------------------
readseq      :       0.196 micros/op;  565.7 MB/s

With patch:

>./db_bench --benchmarks=readseq --num=$((4<<20)) --db=/tmp/db --use_existing_db=1
LevelDB:    version 1.21
Date:       Thu Apr 25 09:38:20 2019
CPU:        32 * Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
CPUCache:   20480 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    4194304
RawSize:    464.0 MB (estimated)
FileSize:   264.0 MB (estimated)
------------------------------------------------
readseq      :       0.181 micros/op;  612.3 MB/s

Signed-off-by: Kyle Zhang <kyle@smartx.com>
2019-04-25 09:54:05 +08:00
leveldb Team
3dc9202f78 [leveldb] Specifically export the WriteBatch::Handler inner class for Windows link
Windows linking visibility in shared libraries requires that inner classes are
specifically exported as visible, even if the containing class is exported.

PiperOrigin-RevId: 244886019
2019-04-23 11:24:04 -07:00
Chris Mumford
2ccb45c33a Check for possibly invalid offset in test.
Fix a possible array bounds offset issue flagged in
issue #668. Not the source of any known bug, but will
silence any static analyzers.

PiperOrigin-RevId: 243697659
2019-04-23 11:23:51 -07:00
Chris Mumford
7b11745190 Changed Windows specific highlighting from bash to cmd.
This makes the syntax highlighting a little nicer on GitHub.

PiperOrigin-RevId: 243426806
2019-04-13 09:20:48 -07:00
Chris Mumford
2f008ac19e Initialize class members to default values in constructors.
There were a few members which were identified to have been left
uninitialized in some constructors. These were very likely to
have been set before being used, otherwise the ASan tests would
have caught them, but still good practice to have them
initialized. This addresses some items reported in issue #668.

PiperOrigin-RevId: 243370145
2019-04-12 18:50:15 -07:00
Chris Mumford
ffabb1ae86 Merge pull request #665 from cheng-chang:coding
PiperOrigin-RevId: 243316002
2019-04-12 13:22:46 -07:00
Chris Mumford
7da571cf2b Merge pull request #669 from pavel-pimenov:fix-readme-windows-mkdir
PiperOrigin-RevId: 243314673
2019-04-12 13:22:36 -07:00
Chris Mumford
df4a323aaf Merge pull request #472 from zhoudayang:patch-1
PiperOrigin-RevId: 243314507
2019-04-12 13:22:24 -07:00
Chris Mumford
5a2a472741 Fixed missing std namespaces and make_unique.
cout/endl were missing the std namespace. Also std::make_unique
was used inadvertently which is part of C++14 and only C++11
is currently supported.

PiperOrigin-RevId: 243221310
2019-04-12 01:11:25 -07:00
Chris Mumford
08e771901f Simplify issue320_test.
Use std::unique_ptr to simplify issue320_test.

PiperOrigin-RevId: 243190799
2019-04-12 00:17:13 -07:00
Chris Mumford
65e86f75ea Fix formatting of recent snapshot compaction fix.
Fix variable names, line lengths, namespace use, and a few other
minor issues to conform to Google C++ style guide.

PiperOrigin-RevId: 243187729
2019-04-12 00:17:03 -07:00
Chris Mumford
7711e76766 Merge pull request #339 from richcole-at-amazon:master
PiperOrigin-RevId: 243156105
2019-04-12 00:16:52 -07:00
Chris Mumford
71ed7c401e Fixed typo in comment in version_set.h.
Flagged by presubmit check.

PiperOrigin-RevId: 243118632
2019-04-12 00:16:41 -07:00
Victor Costan
09fa8868db Align version/soversion CMake setup closer with other repositories.
PiperOrigin-RevId: 241432456
2019-04-01 17:37:45 -07:00
Richard Cole
20fb601aa9 Fix snapshot compaction bug
Closes google/leveldb#320

During compaction it was possible that records from a block b1=(l1,u1)
would be pushed down from level i to level i+1. If there is a block
b2=(l2,u2) at level i with k1 = user_key(u1) = user_key(l2) then
a subsequent search for k1 will yield the record l2 which has a smaller
sequence number than u1 because the sort order for records sorts
increasing by user key but decreaing by sequence number.

This change add a call to a new function AddBoundaryInputs to
SetupOtherInputs. AddBoundaryInputs searches for a block b2 matching the
criteria above and adds it to the set of files to be compacted. Whenever
AddBoundaryInputs is called it is important that the compaction fileset
in level i+1 (known as c->inputs_[1] in the code) be recomputed. Each
call to AddBoundaryInputs is followed by a call to GetOverlappingInputs.

SetupOtherInputs is called on both manual and automated compaction
passes. It is called for both level zero and for levels greater than 0.

The original change posted in https://github.com/google/leveldb/pull/339
has been modified to also include changed made by Chris Mumford<cmumford@google.com>
in 4b72cb14f8

  1. Releasing snapshots during test cleanup to avoid
     memory leak warnings.
  2. Refactored test to use testutil.h to be in line
     with other issue tests and to create the test
     database in the correct temporary location.
  3. Added copyright banner.

  Otherwise, just minor formatting and limiting character
  width to 80 characters.

Additionally the change was rebased on top of current master and
changes previously made to the Makefile were ported to the
CMakeLists.txt.

Testing Done:

  A test program (issue320_test) was constructed that performs mutations
  while snapshots are active. issue320_test fails without this bug fix
  after 64k writes. It passes with this bug fix. It was run with 200M
  writes and passed.

  Unit tests were written for the new function that was added to the
  code. Make test was run and seen to pass.

Signed-off-by: Richard Cole <richcole@amazon.com>
2019-04-01 13:10:09 -07:00
leveldb Team
37300aa54b Restore soname versioning with CMake build
Before:

$ readelf -d build/libleveldb.so | grep soname
 0x000000000000000e (SONAME)             Library soname: [libleveldb.so]

After:
$ readelf -d build/libleveldb.so | grep soname
 0x000000000000000e (SONAME)             Library soname: [libleveldb.so.1]

This matches the soname from v1.20.

PiperOrigin-RevId: 241334113
2019-04-01 09:04:40 -07:00
Pavel Pimenov
952be04df6 Fix mkdir (windows) 2019-03-31 10:46:31 +03:00
Chris Mumford
56178ddaf4 Update the version to 1.21 in preparation for a new release.
PiperOrigin-RevId: 241053616
2019-03-29 14:37:39 -07:00
leveldb Team
35619d248d Project import generated by Copybara.
PiperOrigin-RevId: 241045448
2019-03-29 13:59:02 -07:00
costan
416344de2f leveldb: Register in copybara whitelist.
The documentation recommends modifying the whitelist after evaluating Copybara. However, evaluating requires significant workarounds without the whitelist entry. So, this CL adds leveldb to the whitelist early.

leveldb is currently open sourced to https://github.com/google/leveldb using MOE.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=240786286
2019-03-29 11:22:22 -07:00
costan
da94ac67e9 leveldb: Minor cleanup in ports.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=240619768
2019-03-29 11:22:22 -07:00
costan
bd24b96306 leveldb: Silence unused argument warnings in MSVC.
This CL uses a well-known workaround for silencing arguments that may be unused, depending on the build configuration. The silenced warnings were responsible for a large amount of noise in the MSVC build on Windows.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=240357359
2019-03-29 11:22:22 -07:00
costan
6188a54ce9 leveldb: Add tests for empty keys and values.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=239695281
2019-03-29 11:22:22 -07:00
Cheng Chang
cf1b5f4732 Remove unnecessary bit operation. 2019-03-22 17:32:20 +08:00