Commit Graph

134 Commits

Author SHA1 Message Date
Victor Costan
3ab94e7da8
Merge branch 'master' into fix_issue_505 2022-01-10 09:48:52 -08:00
Victor Costan
8ccb79b57e Merge pull request #901 from mapleFU:opt-using-move
PiperOrigin-RevId: 420662938
2022-01-10 06:35:14 +00:00
Victor Costan
8f5aa6375e Merge pull request #919 from wineway:fix_posix_test
PiperOrigin-RevId: 420644954
2022-01-10 02:29:08 +00:00
Victor Costan
e4ccaa0c9c Merge pull request #965 from ShawnZhong:cpp20
PiperOrigin-RevId: 420504266
2022-01-08 20:55:42 +00:00
Dimitris Apostolou
0e8aa26c4e
Fix typos 2022-01-05 11:04:16 +02:00
Victor Costan
8f464e7f68 Remove main() from most tests.
This gives some flexibility to embedders.

Currently, embedders have to build a binary for each test file.

After this CL, embedders can still choose to have a binary for each test
file, by linking each test file with a googletest target that includes
main() (usually "gtest_main"). Embedders can also choose to build a
single binary for almost all test files, and link with a googletest
target that includes main(). The latter is more convenient for projects
that have very few test binaries, like Chromium.

PiperOrigin-RevId: 419470798
2022-01-03 21:05:04 +00:00
Shawn Zhong
7a2f64ed50 Update env_posix.cc 2021-12-30 18:33:55 -06:00
Victor Costan
335876a133 Add invariant checks to Limiter in Env implementations.
PiperOrigin-RevId: 417853172
2021-12-22 19:26:31 +00:00
Victor Costan
e426c83e88 Merge pull request #941 from pmmp:no-handle-inheritance
PiperOrigin-RevId: 412997201
2021-11-30 00:09:27 +00:00
Felipe Oliveira Carvalho
dd6658754f Remove <pthread.h> include and find_package() from build files 2021-11-28 20:59:14 +01:00
Dylan K. Taylor
68d14a723a
Prevent handle used for LOG from being inherited by subprocesses
I recently encountered a problem with this because Windows doesn't allow
files to be deleted when there's open handles to them.

Other files opened by leveldb are not affected because by and large they
are using CreateFileA, which does not allow inheritance when
lpSecurityAttributes is null (ref:
https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilea)

However, fopen() _does_ allow inheritance, and it needs to be expressly
disabled.
https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/fopen-wfopen?view=msvc-160
2021-10-09 16:22:08 +01:00
leveldb Team
c5d5174a66 Get env_posix.cc building under Fuchsia.
PiperOrigin-RevId: 395824737
2021-09-12 14:12:56 +00:00
wineway
8949158f5d fixed random access file exhaust random mmap file use wrong limit count 2021-07-01 20:52:01 +08:00
mwish
1ca4f5b466 [Init] initial commit 2021-05-02 14:16:51 +08:00
leveldb Team
b7d3023269 Internal cleanup migrating StatusOr.
PiperOrigin-RevId: 329720018
2020-10-07 21:15:26 +00:00
Victor Costan
5c6dd75897 Fix accidental double std:: qualifiers.
PiperOrigin-RevId: 309136120
2020-04-30 01:20:50 +00:00
Victor Costan
a6b3a2012e Add some std:: qualifiers to types and functions.
PiperOrigin-RevId: 309110431
2020-04-29 22:33:14 +00:00
Victor Costan
3f934e3705 Switch from C headers to C++ headers.
This CL makes the following substitutions.

* assert.h -> cassert
* math.h -> cmath
* stdarg.h -> cstdarg
* stddef.h -> cstddef
* stdint.h -> cstdint
* stdio.h -> cstdio
* stdlib.h -> cstdlib
* string.h -> cstring

PiperOrigin-RevId: 309080151
2020-04-29 20:51:13 +00:00
leveldb Team
98a3b8cf65 change const to constexpr
PiperOrigin-RevId: 307113877
2020-04-28 00:17:51 +00:00
Victor Costan
201f52201f Remove leveldb::port::kLittleEndian.
Clang 10 includes the optimizations described in
https://bugs.llvm.org/show_bug.cgi?id=41761. This means that the
platform-independent implementations of {Decode,Encode}Fixed{32,64}()
compile to one instruction on the most recent Clang and GCC.

PiperOrigin-RevId: 306330166
2020-04-14 01:10:05 +00:00
Victor Costan
5903e7a112 Remove Windows workarounds in some tests.
leveldb::Env::DeleteFile was replaced with leveldb::Env::RemoveFile in
all tests. This allows us to remove workarounds for windows.h #defining
DeleteFile.
PiperOrigin-RevId: 289121105
2020-01-14 18:31:37 -08:00
Victor Costan
a0191e5563 Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}.
The "DeleteFile" method name causes pain for Windows developers, because
<windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA.
Current code uses workarounds, like #undefining DeleteFile everywhere an
Env is declared, implemented, or used.

This CL removes the need for workarounds by renaming Env::DeleteFile to
Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to
Env::RemoveDir. A few internal methods are also renamed for consistency.
Software that supports Windows is expected to migrate any Env
implementations and usage to Remove{File,Dir}, and never use the name
Env::Delete{File,Dir} in its code.

The renaming is done in a backwards-compatible way, at the risk of
making it slightly more difficult to build a new correct Env
implementation. The backwards compatibility is achieved using the
following hacks:

1) Env::Remove{File,Dir} methods are added, with a default
    implementation that calls into Env::Delete{File,Dir}. This makes old
    Env implementations compatible with code that calls into the updated
    API.
2) The Env::Delete{File,Dir} methods are no longer pure virtuals.
    Instead, they gain a default implementation that calls into
    Env::Remove{File,Dir}. This makes updated Env implementations
    compatible with code that calls into the old API.

The cost of this approach is that it's possible to write an Env without
overriding either Rename{File,Dir} or Delete{File,Dir}, without getting
a compiler warning. However, attempting to run the test suite will
immediately fail with an infinite call stack ending in
{Remove,Delete}{File,Dir}, making developers aware of the problem.

PiperOrigin-RevId: 288710907
2020-01-09 09:18:14 -08:00
leveldb Team
d152b23f3b Defend against inclusion of windows.h in tests that invoke
Env::DeleteFile.

PiperOrigin-RevId: 283607548
2020-01-09 09:17:59 -08:00
leveldb Team
583a42b596 Internal change.
PiperOrigin-RevId: 282373286
2019-12-02 11:44:39 -08:00
Victor Costan
1c58902bdc Switch testing harness to googletest.
PiperOrigin-RevId: 281815695
2019-11-21 13:11:40 -08:00
Sanjay Ghemawat
60db170a43 Fix tsan problem in env_test.
PiperOrigin-RevId: 268265314
2019-09-29 20:40:29 -07:00
Victor Costan
e0d5f83a4f Align EnvPosix and EnvWindows.
Fixes #695.

PiperOrigin-RevId: 252895299
2019-06-13 13:43:09 -07:00
Victor Costan
72a38ff7f2 Replace "> >" with ">>"
PiperOrigin-RevId: 250383036
2019-05-30 09:55:43 -07:00
Victor Costan
863f185970 unsigned char -> uint8_t
PiperOrigin-RevId: 250309603
2019-05-28 15:44:32 -07:00
Chris Mumford
ae49533210 Add explicit typecasts to avoid compiler warning.
Fixes issue #684.

PiperOrigin-RevId: 249531001
2019-05-24 14:49:37 -07:00
Chris Mumford
28e6d238be Switch to using C++ 11 override specifier.
PiperOrigin-RevId: 247491163
2019-05-09 14:11:06 -07:00
果冻
1d94fe2f4d
Merge branch 'master' into patch-2 2019-05-09 00:08:49 +08:00
Victor Costan
27dc99fb26 Fix EnvPosix tests on Travis CI.
The previous attempt of having EnvPosix use O_CLOEXEC (close-on-exec()) when opening file descriptors added tests that relied on procfs, which is Linux-specific. These tests failed on macOS. Unfortunately, the test failures were not caught due to a (since fixed) error in our Travis CI configuration.

This CL re-structures the tests to only rely on POSIX features. Since there is no POSIX-compliant way to get a file name/path out of a file descriptor, this CL breaks up the O_CLOEXEC test into multiple tests, where each Env method that creates an FD gets its own test. This is intended to make it easier to find and fix errors in Env implementations.

This CL also fixes the implementation of NewLogger() to use O_CLOEXEC on macOS. The current implementation passes "we" to fopen(), but the macOS standard C library does not implement the "e" flag yet.

PiperOrigin-RevId: 247088953
2019-05-07 14:20:31 -07:00
Chris Mumford
9521545b06 Formatting changes for prior O_CLOEXEC fix.
Two minor corrections to correct the 900f7d37eb commit
to conform to the Google C++ style guide.

PiperOrigin-RevId: 246907647
2019-05-06 15:34:20 -07:00
Chris Mumford
900f7d37eb Merge pull request #624 from adam-azarchs:master
PiperOrigin-RevId: 246903086
2019-05-06 15:00:48 -07:00
Victor Costan
a7528a5d2b Clean up util/coding.{h,cc}.
1) Inline EncodeFixed{32,64}(). They emit single machine instructions on 64-bit processors.
2) Remove size narrowing compiler warnings from DecodeFixed{32,64}().
3) Add comments explaining the current state of optimizations in compilers we care about.
4) Change C-style includes, like <stdint.h>, to C++ style, like <cstdint>.
5) memcpy -> std::memcpy.

The optimization comments are based on https://godbolt.org/z/RdIqS1. The missed optimization opportunities in clang have been reported as https://bugs.llvm.org/show_bug.cgi?id=41761

The change does not have significant impact on benchmarks. Results below.

LevelDB:    version 1.22
Date:       Mon May  6 10:42:18 2019
CPU:        72 * Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz
CPUCache:   25344 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    1000000
RawSize:    110.6 MB (estimated)
FileSize:   62.9 MB (estimated)

With change
------------------------------------------------
fillseq      :       2.327 micros/op;   47.5 MB/s
fillsync     :    4185.526 micros/op;    0.0 MB/s (1000 ops)
fillrandom   :       3.662 micros/op;   30.2 MB/s
overwrite    :       4.261 micros/op;   26.0 MB/s
readrandom   :       4.239 micros/op; (1000000 of 1000000 found)
readrandom   :       3.649 micros/op; (1000000 of 1000000 found)
readseq      :       0.174 micros/op;  636.7 MB/s
readreverse  :       0.271 micros/op;  408.7 MB/s
compact      :  570495.000 micros/op;
readrandom   :       2.735 micros/op; (1000000 of 1000000 found)
readseq      :       0.118 micros/op;  937.3 MB/s
readreverse  :       0.190 micros/op;  583.7 MB/s
fill100K     :     860.164 micros/op;  110.9 MB/s (1000 ops)
crc32c       :       1.131 micros/op; 3455.2 MB/s (4K per op)
snappycomp   :       3.034 micros/op; 1287.5 MB/s (output: 55.1%)
snappyuncomp :       0.544 micros/op; 7176.0 MB/s

Baseline
------------------------------------------------
fillseq      :       2.365 micros/op;   46.8 MB/s
fillsync     :    4240.165 micros/op;    0.0 MB/s (1000 ops)
fillrandom   :       3.244 micros/op;   34.1 MB/s
overwrite    :       4.153 micros/op;   26.6 MB/s
readrandom   :       4.698 micros/op; (1000000 of 1000000 found)
readrandom   :       4.065 micros/op; (1000000 of 1000000 found)
readseq      :       0.192 micros/op;  576.3 MB/s
readreverse  :       0.286 micros/op;  386.7 MB/s
compact      :  635979.000 micros/op;
readrandom   :       3.264 micros/op; (1000000 of 1000000 found)
readseq      :       0.169 micros/op;  652.8 MB/s
readreverse  :       0.213 micros/op;  519.5 MB/s
fill100K     :    1055.367 micros/op;   90.4 MB/s (1000 ops)
crc32c       :       1.353 micros/op; 2887.3 MB/s (4K per op)
snappycomp   :       3.036 micros/op; 1286.7 MB/s (output: 55.1%)
snappyuncomp :       0.540 micros/op; 7238.6 MB/s
PiperOrigin-RevId: 246856811
2019-05-06 11:23:02 -07:00
Victor Costan
24424a1ef2 Style cleanup.
1) Convert iterator-based for loops to C++11 foreach loops.
2) Convert "void operator=" to "T& operator=".
3) Switch from copy operators from private to public deleted.
4) Switch from empty ctors / dtors to "= default" where appropriate.

PiperOrigin-RevId: 246679195
2019-05-04 17:42:20 -07:00
Chris Mumford
9bd23c7676 Correct class/structure declaration order.
1. Correct the class/struct declaration order to be IAW
   the Google C++ style guide[1].
2. For non-copyable classes, switched from non-implemented
   private methods to explicitly deleted[2] methods.
3. Minor const and member initialization fixes.

[1] https://google.github.io/styleguide/cppguide.html#Declaration_Order
[2] http://eel.is/c++draft/dcl.fct.def.delete

PiperOrigin-RevId: 246521844
2019-05-03 09:48:57 -07:00
Chris Mumford
297e66afc1 Format all files IAW the Google C++ Style Guide.
Use clang-format to correct formatting to be in agreement with the [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html). Doing this simplifies the process of accepting changes. Also fixed a few warnings flagged by clang-tidy.

PiperOrigin-RevId: 246350737
2019-05-02 19:04:50 -07:00
Chris Mumford
2f008ac19e Initialize class members to default values in constructors.
There were a few members which were identified to have been left
uninitialized in some constructors. These were very likely to
have been set before being used, otherwise the ASan tests would
have caught them, but still good practice to have them
initialized. This addresses some items reported in issue #668.

PiperOrigin-RevId: 243370145
2019-04-12 18:50:15 -07:00
Chris Mumford
ffabb1ae86 Merge pull request #665 from cheng-chang:coding
PiperOrigin-RevId: 243316002
2019-04-12 13:22:46 -07:00
Cheng Chang
cf1b5f4732 Remove unnecessary bit operation. 2019-03-22 17:32:20 +08:00
Felipe Oliveira Carvalho
7035af5fc3 Two small fixes for the Windows implementation (#661)
* Check if NOMIMMAX is defined before defining it

* Pass char* for a %s format in a snprintf call
2019-03-21 08:45:04 -07:00
costan
15e2278966 Use override consistently in leveldb::test::ErrorEnv.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=239453565
2019-03-20 13:57:32 -07:00
cmumford
ea49b27d06 Switch corruption_test to use InMemEnv.
This change switches corruption_test, which previously used direct file
I/O to corrupt table files for open databases, to use InMemEnv. Using an
Env eliminates some platform dependencies thus simplifying the tests.

Also removed EnvWindowsTestHelper::RelaxFilePermissions().  This was
only added because the Windows Env opens files for exclusive access.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=239305329
2019-03-20 13:57:03 -07:00
costan
201f77d137 Inline defaults in options.
This CL moves default values for
leveldb::{Options,ReadOptions,WriteOptions} from constructors to member
declarations, and removes now-redundant comments stating the defaults.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=239271242
2019-03-20 13:56:22 -07:00
costan
7d8e41e49b leveldb: Replace AtomicPointer with std::atomic.
This CL removes AtomicPointer from leveldb's port interface. Its usage is replaced with std::atomic<> from the C++11 standard library.

AtomicPointer was used to wrap flags, numbers, and pointers, so its instances are replaced with std::atomic<bool>, std::atomic<int>, std::atomic<size_t> and std::atomic<Node*>.

This CL does not revise the memory ordering. AtomicPointer's methods are replaced mechanically with their std::atomic equivalents, even when the underlying usage is incorrect. (Example: DBImpl::has_imm_ is written using release stores, even though it is always read using relaxed ordering.) Revising the memory ordering is left for future CLs.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=237865146
2019-03-11 13:41:25 -07:00
costan
ed76289b25 Align windows_logger with posix_logger.
Fixes GitHub issue #657.

This CL also makes the Windows CI green.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=237255887
2019-03-07 10:04:01 -08:00
cmumford
c69d33b0ec Added native support for Windows.
This change adds a native Windows port (port_windows.h) and a
Windows Env (WindowsEnv).

Note1: "small" is defined when including <Windows.h> so some
parameters were renamed to avoid conflict.

Note2: leveldb::Env defines the method: "DeleteFile" which is
also a constant defined when including <Windows.h>. The solution
was to ensure this macro is defined in env.h which forces
the function, when compiled, to be either DeleteFileA or
DeleteFileW when building for MBCS or UNICODE respectively.

This resolves #519 on GitHub.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=236364778
2019-03-01 18:00:35 -08:00
Adam Azarchs
75fceae700 Add O_CLOEXEC to open calls.
This prevents file descriptors from leaking to child processes.

When compiled for older (pre-2.6.23) kernels which lack support for
O_CLOEXEC there is no change in behavior.  With newer kernels, child
processes will no longer inherit leveldb's file handles, which
reduces the changes of accidentally corrupting the database.

Fixes https://github.com/google/leveldb/issues/623
2019-02-22 13:00:56 -08:00