mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-12 00:19:14 +08:00
Code Issues Packages Projects Releases Wiki Activity
9,064 Commits 32 Branches 296 Tags 150 MiB
693e07b509
Commit Graph

678 Commits

Author SHA1 Message Date
Konstantin Lebedev
f523aed3c9 fixes from comments 
https://github.com/chrislusf/seaweedfs/pull/2996
 2022-05-11 21:50:09 +05:00
chrislu
76da688176 adjust log level 2022-05-11 00:45:19 -07:00
chrislu
a85720f0f6 remove deprecated function 2022-05-10 00:02:23 -07:00
chrislu
791c29ae93 remove deprecated function 
WithQueryCachePolicyKeepInCache is deprecated and already default behavior
 2022-05-10 00:01:43 -07:00
chrislu
10207a48a4 how to setup local test env 2022-05-06 02:25:56 -07:00
chrislu
db33b03735 add comments 2022-05-06 02:19:47 -07:00
chrislu
e350ba8896 larger test range 2022-05-06 02:17:28 -07:00
chrislu
71e0bcc5d5 a bit optimization 2022-05-06 02:11:58 -07:00
chrislu
b2a43dd044 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-06 01:51:28 -07:00
chrislu
d5fe5d12b7 filer: etcd fix pagination 
fix https://github.com/chrislusf/seaweedfs/issues/3016
 2022-05-06 01:51:19 -07:00
Konstantin Lebedev
e6d2cb5911 ydb add empty doc file 2022-05-05 11:12:14 +05:00
Konstantin Lebedev
f58adaab25 build tag ydb since the growth of the binary is 5 mb 2022-05-03 23:20:09 +05:00
Konstantin Lebedev
f127b326bf add options to scaffold 2022-05-03 22:54:31 +05:00
Konstantin Lebedev
7b544576af refactor 2022-05-03 20:16:00 +05:00
Konstantin Lebedev
04d5dff6bb add support native ttl 2022-05-03 17:52:23 +05:00
Konstantin Lebedev
cb3c7a3cdb enable query cache policy instead of prepare 2022-05-03 16:03:10 +05:00
Konstantin Lebedev
8342f651f3 fix scanNamed 2022-05-03 15:18:28 +05:00
Konstantin Lebedev
d8925b4e83 Merge branch 'new_master' into ydb 
# Conflicts:
#	go.mod
#	go.sum
 2022-05-03 00:13:57 +05:00
Konstantin Lebedev
7640e650e5 fix queries 2022-05-03 00:11:37 +05:00
Konstantin Lebedev
319d300d48 fix createDB 2022-05-02 22:23:07 +05:00
Konstantin Lebedev
0dc44dda63 ydb do Tx or DB 2022-05-02 15:33:29 +05:00
Konstantin Lebedev
a3e48831f3 ydb BucketAware interface 2022-05-02 12:42:20 +05:00
chrislu
94635e9b5c filer: add filer group 2022-05-01 21:59:16 -07:00
Konstantin Lebedev
50c4f62ed4 ydb-go-sdk move to v3 2022-05-02 02:07:47 +05:00
Konstantin Lebedev
21033ff4c3 refactor use const CountEntryChunksForGzip 2022-05-01 22:28:55 +05:00
Konstantin Lebedev
ec0ed41e37 ydb kv interface 2022-05-01 21:20:37 +05:00
Konstantin Lebedev
6a052f6ff2 ydb Sql interface 2022-05-01 20:33:03 +05:00
Konstantin Lebedev
275c3bb19c ydb initial 
https://github.com/chrislusf/seaweedfs/issues/2942
 2022-05-01 17:28:58 +05:00
chrislu
4aa39ef33f conditionally compile sqlite 2022-04-21 01:22:38 -07:00
chrislu
3885374edf conditionally build elastic, gocdk to reduce binary size 2022-04-21 01:10:46 -07:00
chrislu
cb1e7aa27d fix build 2022-04-20 16:21:56 -07:00
a
1d6a9e66b6 Merge branch 'master' into a 2022-04-20 14:01:42 -07:00
justin
3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
eddy-gfx
ec53eec94f
Merge branch 'master' into a 2022-04-06 18:45:55 -05:00
chrislu
bb4beebce3 prevent nil 2022-04-05 10:49:17 -07:00
a
7e92517571 change user and pass to username and password 2022-04-01 14:09:25 -05:00
a
bc603e534f diff 2022-03-28 16:50:28 +00:00
chrislu
86d4c52d6e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-03-26 13:33:20 -07:00
chrislu
21e0898631 refactor: change masters from a slice to a map 2022-03-26 13:33:17 -07:00
Chris Lu
3153c836a7
Merge pull request #2744 from kmlebedev/issues_2545 
avoid connect to the old filler address
 2022-03-26 13:02:49 -07:00
chrislu
cae998eda1 Revert "Revert "remove duplicated metadata subscription in filer"" 
This reverts commit 34b743c481.
 2022-03-26 13:00:16 -07:00
chrislu
34b743c481 Revert "remove duplicated metadata subscription in filer" 
This reverts commit 34742be029.

Related to https://github.com/chrislusf/seaweedfs/issues/2545
 2022-03-26 12:33:45 -07:00
chrislu
4e2388e1b5 mount: set file size if it is only on remote gateway 2022-03-21 02:09:54 -07:00
elee
beb406bbbb fix ls, onBucketDelete still not triggering 2022-03-18 22:21:57 -05:00
elee
411c0df3fe switch to multi collection, change readme 2022-03-18 21:51:16 -05:00
elee
1cea6c73d3 update readme 2022-03-18 00:34:19 -05:00
elee
25be96832a update readme with index info 2022-03-18 00:33:08 -05:00
elee
06f23aa675 put in delete folder children query 2022-03-18 00:29:48 -05:00
elee
0701feeb17 added more to readme 2022-03-18 00:26:11 -05:00
elee
171c27ddf1 zz 2022-03-17 22:35:30 -05:00
elee
2f0cdcdceb update readme 2022-03-17 21:26:16 -05:00
elee
423ce57cde prefix search, bucket implemented 2022-03-17 21:12:25 -05:00
elee
bf745bdccb revise comment 2022-03-17 14:09:34 -05:00
elee
a0df993cef cleanup, add a few comments 2022-03-17 14:08:36 -05:00
elee
94883e2fad ok then 2022-03-17 05:23:12 -05:00
elee
f3ab6769e9 make start from indexing work 2022-03-17 05:09:13 -05:00
elee
31571dd96e fix file listing 2022-03-17 05:07:40 -05:00
elee
b25d03340b Merge branch 'a' of github.com:gfxlabs/seaweedfs into a 2022-03-17 04:51:59 -05:00
elee
921535001a arangodb adapter 2022-03-17 04:49:26 -05:00
chrislu
6886167d5f minor optimization 2022-03-17 00:03:28 -07:00
chrislu
4042fdf3bb rename to skipCheckParentDir 
related to https://github.com/chrislusf/seaweedfs/pull/2761

It's better to default to false.
 2022-03-16 23:55:31 -07:00
zzq09494
40b0033fa7 go fmt 2022-03-17 14:19:48 +08:00
zzq09494
ee5417a5b3 fix: ut test error 2022-03-17 11:16:46 +08:00
zzq09494
81cce4b4c3 filer: support uploading file without needEnsureParentDir 2022-03-17 10:53:47 +08:00
zzq09494
a6a8892255 Revert "filer: support uploading file without needEnsureParentDir" 
This reverts commit a93c4947ba.
 2022-03-17 10:27:17 +08:00
zzq09494
a93c4947ba filer: support uploading file without needEnsureParentDir 2022-03-17 10:18:23 +08:00
chrislu
fbc9f0eb64 minor 2022-03-14 03:19:16 -07:00
chrislu
2f4cd20f90 tests skip reader pattern monitoring 2022-03-13 18:15:53 -07:00
chrislu
53513475bf mount: add back random read support 
avoid too much memory used also
 2022-03-13 01:38:52 -08:00
Konstantin Lebedev
834210a9dc avoid connect to the old filler address 
https://github.com/chrislusf/seaweedfs/issues/2545
 2022-03-10 15:24:45 +05:00
zzq09494
2ea18fca48 fix:support some databases which not support 'IF NOT EXISTS' keyword of 'create table command' 2022-03-10 13:37:34 +08:00
banjiaojuhao
f7f2a597dd minor 2022-03-08 16:22:55 +08:00
banjiaojuhao
d61bea9038 [bugfix] filer: In file modification, old chunks will be mis-deleted when they are merged(Manifestized). 2022-03-08 16:22:55 +08:00
chrislu
6f3ec989a7 fix manifest batch size 2022-03-07 00:07:53 -08:00
banjiaojuhao
a07c93f4dd filer: reset bytesBuffer before use 2022-03-06 21:07:36 +08:00
chrislu
da76af187f mount: avoid possible index out of bounds error 2022-03-04 22:36:01 -08:00
chrislu
6fbbc78574 stream reading a whole chunk 2022-03-02 13:50:46 -08:00
chrislu
941ced60a4 download 2 chunks if at the beginning of a file 2022-02-27 03:57:24 -08:00
chrislu
551d00d51a prefetch other chunks when stream reading 2022-02-26 23:20:45 -08:00
chrislu
7b1a713d2a remove dead code 2022-02-26 03:23:15 -08:00
chrislu
3a58b7bac6 a little safer 2022-02-26 03:23:06 -08:00
chrislu
3345a50d9b prefetch 2 chunks 2022-02-26 03:06:17 -08:00
chrislu
86ce69f709 remove logs 2022-02-26 03:00:20 -08:00
chrislu
28b395bef4 better control for reader caching 2022-02-26 02:16:47 -08:00
chrislu
72c0233938 less logs 2022-02-25 15:34:24 -08:00
chrislu
03466f955e rename: delete source entry metadata only, skipping hard links 2022-02-25 02:57:54 -08:00
chrislu
8080fe4cc1 logs 2022-02-25 02:56:23 -08:00
chrislu
202a29d014 refactoring 2022-02-25 01:17:26 -08:00
chrislu
be3fc77391 mount2: use consistent inode 2022-02-25 00:53:27 -08:00
chrislu
ceaf993a27 mount2: add rdev 2022-02-24 14:51:25 -08:00
chrislu
b05962b90e rename: handle hard links 2022-02-24 02:59:00 -08:00
chrislu
56da3494cb leveldb3: adjust memory allocation for each bucket 2022-02-20 13:40:13 -08:00
chrislu
7bc67399e4 listing for filer stores without prefixed query: break if no more progress 2022-02-19 10:12:51 -08:00
chrislu
248c0c8087 minor 2022-02-19 09:54:09 -08:00
chrislu
4ee0a6f47b filer store: reduce one possible listing operation 2022-02-19 00:43:42 -08:00
chrislu
b3594278c9 optimize a bit 2022-02-18 23:36:10 -08:00
chrislu
df51e0c042 mongodb: remove ErrNoDocuments checking since it is only for creating single record, not for querying 2022-02-15 16:18:36 -08:00
chrislu
98cce6a150 mongodb: ensure closing cursor 2022-02-15 16:15:44 -08:00
banjiaojuhao
b5ec346700 FilerStore: add redis_lua 2022-02-15 20:54:57 +08:00
Eng Zer Jun
b92df1654c
test: use T.TempDir to create temporary test directory 
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2022-02-14 10:38:13 +08:00
chrislu
85c1615b43 filer read empty file may cause OOM in some cases 
fix https://github.com/chrislusf/seaweedfs/issues/2641
 2022-02-07 23:08:54 -08:00
chrislu
9405eaefdb filer.sync: fix replicating partially updated file 
Run two servers with volumes and fillers:
server -dir=Server1alpha -master.port=11000 -filer -filer.port=11001 -volume.port=11002
server -dir=Server1sigma -master.port=11006 -filer -filer.port=11007 -volume.port=11008

Run Active-Passive filler.sync:
filer.sync -a localhost:11007 -b localhost:11001 -isActivePassive

Upload file to 11007 port:
curl -F file=@/Desktop/9.xml "http://localhost:11007/testFacebook/"

If we request a file on two servers now, everything will be correct, even if we add data to the file and upload it again:
curl "http://localhost:11007/testFacebook/9.xml"
EQUALS
curl "http://localhost:11001/testFacebook/9.xml"

However, if we change the already existing data in the file (for example, we change the first line in the file, reducing its length), then this file on the second server will not be valid and will not be equivalent to the first file

Снимок экрана 2022-02-07 в 14 21 11

This problem occurs on line 202 in the filer_sink.go file. In particular, this is due to incorrect mapping of chunk names in the DoMinusChunks function. The names of deletedChunks do not match the chunks of existingEntry.Chunks, since the first chunks come from another server and have a different addressing (name) compared to the addressing on the server where the file is being overwritten.

Deleted chunks are not actually deleted on the server to which the file is replicated.
 2022-02-07 03:46:28 -08:00
chrislu
4a311c7f5e dedup local metadata subscribers 
fix https://github.com/chrislusf/seaweedfs/discussions/2542
 2022-01-23 16:14:22 -08:00
chrislu
606667f205 able to configure the quota for a bucket 2022-01-21 01:42:20 -08:00
guol-fnst
da9540e666 add gocql timeout setting 2022-01-18 15:21:13 +08:00
banjiaojuhao
45e9c83421 padding zero for sparse file 2022-01-13 22:21:22 +08:00
chrislu
1453263b63 remove dead code 2022-01-13 02:02:04 -08:00
chrislu
10ecf80ca1 add a debug capability to list all metadata keys 2022-01-11 23:25:04 -08:00
chrislu
34742be029 remove duplicated metadata subscription in filer 
https://github.com/chrislusf/seaweedfs/issues/2545
 2021-12-30 01:51:52 -08:00
chrislu
9a00c17555 reader: avoid wrong pattern detection due to lock waiting 2021-12-28 16:30:33 -08:00
chenkai
47c30e3add filer list entries use context to break job 2021-12-28 15:03:41 +08:00
chrislu
9f9ef1340c use streaming mode for long poll grpc calls 
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
 2021-12-26 00:15:03 -08:00
banjiaojuhao
083bf3a137 filer server: add "datacenter, rack and datanode" for path specific configuration 2021-12-23 23:25:05 +08:00
chrislu
b541e39a2c fix tests 2021-12-22 16:17:30 -08:00
chrislu
0ec7bc6710 detect non streaming mode on the first read 2021-12-22 16:05:38 -08:00
chrislu
4c1368d621 fix test 2021-12-22 16:05:08 -08:00
chrislu
0cb9036f66 mount: only cache the first chunk on stream read 2021-12-19 23:06:03 -08:00
chrislu
a152f17937 mount: improve read performance on random reads 2021-12-19 22:43:14 -08:00
Bl1tz23
83bffca7e6 fix MongodbStore.ListDirectoryEntries panic on find failure 2021-12-03 13:58:37 +03:00
Chris Lu
689f5513a9 redis3 supports sentinel 2021-11-29 01:09:51 -08:00
Chris Lu
2f72c24498 skip the rest logic 2021-11-29 00:32:21 -08:00
Chris Lu
96c66ca2aa read deleted chunks when replicating data 2021-11-28 23:33:03 -08:00
limd
ec03f22cc3 Merge remote-tracking branch 'origin/master' 2021-11-25 16:07:14 +08:00
limd
220797bd71 support redis sentinel 2021-11-25 15:57:03 +08:00
Chris Lu
7bf891c00a randomize same-dc servers and other-dc servers 2021-11-12 11:30:11 -08:00
Chris Lu
4729a57cc0 use constants 2021-11-08 17:47:56 -08:00
Chris Lu
e0fc2898e9 auto updated filer peer list 2021-11-06 14:23:35 -07:00
Chris Lu
5ea86ef1da Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()" 
This reverts commit af71ae11aa.
 2021-11-05 17:52:15 -07:00
Chris Lu
7348d2ae63 randomize traffic if replicated 2021-11-05 00:03:13 -07:00
Chris Lu
af71ae11aa master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates() 2021-11-03 01:09:48 -07:00
Chris Lu
0c8dea9de8 go fmt 2021-11-02 23:39:16 -07:00
Chris Lu
1b90d60765 filer: paginate large directories to delete file chunks 2021-11-02 01:04:50 -07:00
Chris Lu
24858507cc rename API to avoid confusion 2021-10-30 19:27:25 -07:00
Chris Lu
f5824f369c avoid reporting error when it is a user request error 2021-10-29 14:14:29 -07:00
Chris Lu
2e76834e4d filer store: redis2 fix wrong pagination 2021-10-27 23:46:25 -07:00
Chris Lu
900e888695 filer: avoid possible prefixed pagination loop on unsupported filer stores 2021-10-27 23:45:48 -07:00
Chris Lu
8e2c9713a3 turn on new faster algorithm to translate into visible chunks 2021-10-16 23:35:20 -07:00
Chris Lu
e7d1f183cc ensure file id is populated 2021-10-16 23:34:27 -07:00
Chris Lu
fc1694b321 add tests 2021-10-16 23:34:12 -07:00
Chris Lu
b9a2efd69b temporarily reverting 2021-10-16 16:29:10 -07:00
Chris Lu
8a124ef9fc Revert "remove deprecated code" 
This reverts commit de7688c539.
 2021-10-16 16:09:33 -07:00
Chris Lu
93624635b4 Revert "go fmt" 
This reverts commit 6c704eb2ba.
 2021-10-16 16:09:30 -07:00
Chris Lu
6c704eb2ba go fmt 2021-10-16 16:06:18 -07:00
Chris Lu
de7688c539 remove deprecated code 2021-10-16 16:06:12 -07:00
Chris Lu
7336990639 faster file read for large files 2021-10-16 16:03:16 -07:00
Chris Lu
8e3075d051 ensure file entry chunks have populated the file id 2021-10-16 01:05:48 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-14 12:27:58 +08:00
Konstantin Lebedev
9d6ffa0ea1 GetBucketLifecycleConfigurationHandler 2021-10-14 01:35:33 +05:00
Konstantin Lebedev
be4b3ed509 AclHandlers 2021-10-11 15:03:56 +05:00
Chris Lu
f4676824a7 fix test code compilation 2021-10-10 22:14:13 -07:00