mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-11 15:59:06 +08:00
Code Issues Packages Projects Releases Wiki Activity
8,404 Commits 32 Branches 296 Tags 150 MiB
5c6c1e7e30
Commit Graph

626 Commits

Author SHA1 Message Date
chrislu
a85720f0f6 remove deprecated function 2022-05-10 00:02:23 -07:00
chrislu
791c29ae93 remove deprecated function 
WithQueryCachePolicyKeepInCache is deprecated and already default behavior
 2022-05-10 00:01:43 -07:00
chrislu
10207a48a4 how to setup local test env 2022-05-06 02:25:56 -07:00
chrislu
db33b03735 add comments 2022-05-06 02:19:47 -07:00
chrislu
e350ba8896 larger test range 2022-05-06 02:17:28 -07:00
chrislu
71e0bcc5d5 a bit optimization 2022-05-06 02:11:58 -07:00
chrislu
b2a43dd044 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-06 01:51:28 -07:00
chrislu
d5fe5d12b7 filer: etcd fix pagination 
fix https://github.com/chrislusf/seaweedfs/issues/3016
 2022-05-06 01:51:19 -07:00
Konstantin Lebedev
e6d2cb5911 ydb add empty doc file 2022-05-05 11:12:14 +05:00
Konstantin Lebedev
f58adaab25 build tag ydb since the growth of the binary is 5 mb 2022-05-03 23:20:09 +05:00
Konstantin Lebedev
f127b326bf add options to scaffold 2022-05-03 22:54:31 +05:00
Konstantin Lebedev
7b544576af refactor 2022-05-03 20:16:00 +05:00
Konstantin Lebedev
04d5dff6bb add support native ttl 2022-05-03 17:52:23 +05:00
Konstantin Lebedev
cb3c7a3cdb enable query cache policy instead of prepare 2022-05-03 16:03:10 +05:00
Konstantin Lebedev
8342f651f3 fix scanNamed 2022-05-03 15:18:28 +05:00
Konstantin Lebedev
d8925b4e83 Merge branch 'new_master' into ydb 
# Conflicts:
#	go.mod
#	go.sum
 2022-05-03 00:13:57 +05:00
Konstantin Lebedev
7640e650e5 fix queries 2022-05-03 00:11:37 +05:00
Konstantin Lebedev
319d300d48 fix createDB 2022-05-02 22:23:07 +05:00
Konstantin Lebedev
0dc44dda63 ydb do Tx or DB 2022-05-02 15:33:29 +05:00
Konstantin Lebedev
a3e48831f3 ydb BucketAware interface 2022-05-02 12:42:20 +05:00
chrislu
94635e9b5c filer: add filer group 2022-05-01 21:59:16 -07:00
Konstantin Lebedev
50c4f62ed4 ydb-go-sdk move to v3 2022-05-02 02:07:47 +05:00
Konstantin Lebedev
21033ff4c3 refactor use const CountEntryChunksForGzip 2022-05-01 22:28:55 +05:00
Konstantin Lebedev
ec0ed41e37 ydb kv interface 2022-05-01 21:20:37 +05:00
Konstantin Lebedev
6a052f6ff2 ydb Sql interface 2022-05-01 20:33:03 +05:00
Konstantin Lebedev
275c3bb19c ydb initial 
https://github.com/chrislusf/seaweedfs/issues/2942
 2022-05-01 17:28:58 +05:00
chrislu
4aa39ef33f conditionally compile sqlite 2022-04-21 01:22:38 -07:00
chrislu
3885374edf conditionally build elastic, gocdk to reduce binary size 2022-04-21 01:10:46 -07:00
chrislu
cb1e7aa27d fix build 2022-04-20 16:21:56 -07:00
a
1d6a9e66b6 Merge branch 'master' into a 2022-04-20 14:01:42 -07:00
justin
3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
eddy-gfx
ec53eec94f
Merge branch 'master' into a 2022-04-06 18:45:55 -05:00
chrislu
bb4beebce3 prevent nil 2022-04-05 10:49:17 -07:00
a
7e92517571 change user and pass to username and password 2022-04-01 14:09:25 -05:00
a
bc603e534f diff 2022-03-28 16:50:28 +00:00
chrislu
86d4c52d6e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-03-26 13:33:20 -07:00
chrislu
21e0898631 refactor: change masters from a slice to a map 2022-03-26 13:33:17 -07:00
Chris Lu
3153c836a7
Merge pull request #2744 from kmlebedev/issues_2545 
avoid connect to the old filler address
 2022-03-26 13:02:49 -07:00
chrislu
cae998eda1 Revert "Revert "remove duplicated metadata subscription in filer"" 
This reverts commit 34b743c481.
 2022-03-26 13:00:16 -07:00
chrislu
34b743c481 Revert "remove duplicated metadata subscription in filer" 
This reverts commit 34742be029.

Related to https://github.com/chrislusf/seaweedfs/issues/2545
 2022-03-26 12:33:45 -07:00
chrislu
4e2388e1b5 mount: set file size if it is only on remote gateway 2022-03-21 02:09:54 -07:00
elee
beb406bbbb fix ls, onBucketDelete still not triggering 2022-03-18 22:21:57 -05:00
elee
411c0df3fe switch to multi collection, change readme 2022-03-18 21:51:16 -05:00
elee
1cea6c73d3 update readme 2022-03-18 00:34:19 -05:00
elee
25be96832a update readme with index info 2022-03-18 00:33:08 -05:00
elee
06f23aa675 put in delete folder children query 2022-03-18 00:29:48 -05:00
elee
0701feeb17 added more to readme 2022-03-18 00:26:11 -05:00
elee
171c27ddf1 zz 2022-03-17 22:35:30 -05:00
elee
2f0cdcdceb update readme 2022-03-17 21:26:16 -05:00
elee
423ce57cde prefix search, bucket implemented 2022-03-17 21:12:25 -05:00
elee
bf745bdccb revise comment 2022-03-17 14:09:34 -05:00
elee
a0df993cef cleanup, add a few comments 2022-03-17 14:08:36 -05:00
elee
94883e2fad ok then 2022-03-17 05:23:12 -05:00
elee
f3ab6769e9 make start from indexing work 2022-03-17 05:09:13 -05:00
elee
31571dd96e fix file listing 2022-03-17 05:07:40 -05:00
elee
b25d03340b Merge branch 'a' of github.com:gfxlabs/seaweedfs into a 2022-03-17 04:51:59 -05:00
elee
921535001a arangodb adapter 2022-03-17 04:49:26 -05:00
chrislu
6886167d5f minor optimization 2022-03-17 00:03:28 -07:00
chrislu
4042fdf3bb rename to skipCheckParentDir 
related to https://github.com/chrislusf/seaweedfs/pull/2761

It's better to default to false.
 2022-03-16 23:55:31 -07:00
zzq09494
40b0033fa7 go fmt 2022-03-17 14:19:48 +08:00
zzq09494
ee5417a5b3 fix: ut test error 2022-03-17 11:16:46 +08:00
zzq09494
81cce4b4c3 filer: support uploading file without needEnsureParentDir 2022-03-17 10:53:47 +08:00
zzq09494
a6a8892255 Revert "filer: support uploading file without needEnsureParentDir" 
This reverts commit a93c4947ba.
 2022-03-17 10:27:17 +08:00
zzq09494
a93c4947ba filer: support uploading file without needEnsureParentDir 2022-03-17 10:18:23 +08:00
chrislu
fbc9f0eb64 minor 2022-03-14 03:19:16 -07:00
chrislu
2f4cd20f90 tests skip reader pattern monitoring 2022-03-13 18:15:53 -07:00
chrislu
53513475bf mount: add back random read support 
avoid too much memory used also
 2022-03-13 01:38:52 -08:00
Konstantin Lebedev
834210a9dc avoid connect to the old filler address 
https://github.com/chrislusf/seaweedfs/issues/2545
 2022-03-10 15:24:45 +05:00
zzq09494
2ea18fca48 fix:support some databases which not support 'IF NOT EXISTS' keyword of 'create table command' 2022-03-10 13:37:34 +08:00
banjiaojuhao
f7f2a597dd minor 2022-03-08 16:22:55 +08:00
banjiaojuhao
d61bea9038 [bugfix] filer: In file modification, old chunks will be mis-deleted when they are merged(Manifestized). 2022-03-08 16:22:55 +08:00
chrislu
6f3ec989a7 fix manifest batch size 2022-03-07 00:07:53 -08:00
banjiaojuhao
a07c93f4dd filer: reset bytesBuffer before use 2022-03-06 21:07:36 +08:00
chrislu
da76af187f mount: avoid possible index out of bounds error 2022-03-04 22:36:01 -08:00
chrislu
6fbbc78574 stream reading a whole chunk 2022-03-02 13:50:46 -08:00
chrislu
941ced60a4 download 2 chunks if at the beginning of a file 2022-02-27 03:57:24 -08:00
chrislu
551d00d51a prefetch other chunks when stream reading 2022-02-26 23:20:45 -08:00
chrislu
7b1a713d2a remove dead code 2022-02-26 03:23:15 -08:00
chrislu
3a58b7bac6 a little safer 2022-02-26 03:23:06 -08:00
chrislu
3345a50d9b prefetch 2 chunks 2022-02-26 03:06:17 -08:00
chrislu
86ce69f709 remove logs 2022-02-26 03:00:20 -08:00
chrislu
28b395bef4 better control for reader caching 2022-02-26 02:16:47 -08:00
chrislu
72c0233938 less logs 2022-02-25 15:34:24 -08:00
chrislu
03466f955e rename: delete source entry metadata only, skipping hard links 2022-02-25 02:57:54 -08:00
chrislu
8080fe4cc1 logs 2022-02-25 02:56:23 -08:00
chrislu
202a29d014 refactoring 2022-02-25 01:17:26 -08:00
chrislu
be3fc77391 mount2: use consistent inode 2022-02-25 00:53:27 -08:00
chrislu
ceaf993a27 mount2: add rdev 2022-02-24 14:51:25 -08:00
chrislu
b05962b90e rename: handle hard links 2022-02-24 02:59:00 -08:00
chrislu
56da3494cb leveldb3: adjust memory allocation for each bucket 2022-02-20 13:40:13 -08:00
chrislu
7bc67399e4 listing for filer stores without prefixed query: break if no more progress 2022-02-19 10:12:51 -08:00
chrislu
248c0c8087 minor 2022-02-19 09:54:09 -08:00
chrislu
4ee0a6f47b filer store: reduce one possible listing operation 2022-02-19 00:43:42 -08:00
chrislu
b3594278c9 optimize a bit 2022-02-18 23:36:10 -08:00
chrislu
df51e0c042 mongodb: remove ErrNoDocuments checking since it is only for creating single record, not for querying 2022-02-15 16:18:36 -08:00
chrislu
98cce6a150 mongodb: ensure closing cursor 2022-02-15 16:15:44 -08:00
banjiaojuhao
b5ec346700 FilerStore: add redis_lua 2022-02-15 20:54:57 +08:00
Eng Zer Jun
b92df1654c
test: use T.TempDir to create temporary test directory 
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2022-02-14 10:38:13 +08:00
chrislu
85c1615b43 filer read empty file may cause OOM in some cases 
fix https://github.com/chrislusf/seaweedfs/issues/2641
 2022-02-07 23:08:54 -08:00
chrislu
9405eaefdb filer.sync: fix replicating partially updated file 
Run two servers with volumes and fillers:
server -dir=Server1alpha -master.port=11000 -filer -filer.port=11001 -volume.port=11002
server -dir=Server1sigma -master.port=11006 -filer -filer.port=11007 -volume.port=11008

Run Active-Passive filler.sync:
filer.sync -a localhost:11007 -b localhost:11001 -isActivePassive

Upload file to 11007 port:
curl -F file=@/Desktop/9.xml "http://localhost:11007/testFacebook/"

If we request a file on two servers now, everything will be correct, even if we add data to the file and upload it again:
curl "http://localhost:11007/testFacebook/9.xml"
EQUALS
curl "http://localhost:11001/testFacebook/9.xml"

However, if we change the already existing data in the file (for example, we change the first line in the file, reducing its length), then this file on the second server will not be valid and will not be equivalent to the first file

Снимок экрана 2022-02-07 в 14 21 11

This problem occurs on line 202 in the filer_sink.go file. In particular, this is due to incorrect mapping of chunk names in the DoMinusChunks function. The names of deletedChunks do not match the chunks of existingEntry.Chunks, since the first chunks come from another server and have a different addressing (name) compared to the addressing on the server where the file is being overwritten.

Deleted chunks are not actually deleted on the server to which the file is replicated.
 2022-02-07 03:46:28 -08:00