chrislu
860c207c94
typo
2022-07-12 02:56:19 -07:00
guosj
5469efa336
close SkipCheckParentDirectory flag
2022-07-11 09:44:48 +08:00
chrislu
185332b91f
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-07-08 00:29:41 -07:00
chrislu
28add5a534
mount: fix racing conditions
...
prevent wrong reading when the SingleChunkCacher is started, but not finished yet
2022-07-08 00:29:39 -07:00
duanhongyi
1ceab96aba
filer tikv support tls
2022-07-08 14:23:06 +08:00
chrislu
a85ed3fe8f
minor
2022-07-07 22:27:24 -07:00
chrislu
2ea18cdcc8
remove dead code
2022-07-07 22:26:03 -07:00
chrislu
f2f0482dd3
mount: random read also try to use the local cache first
2022-07-07 11:50:28 -07:00
石昌林
2b5b37231b
Make sure that onChangeEvent is called successfully before updating lastTsNs
2022-06-29 23:57:07 +08:00
石昌林
e17e6ed542
Avoid synchronizing from scratch due to err
2022-06-29 23:31:04 +08:00
yulai.li
824cbe32be
Make tikv filer enable/disable by build tags
2022-06-27 10:57:24 +08:00
yulai.li
46e0b629e5
Update tikv client version and add one PC support
2022-06-26 22:43:37 +08:00
LHHDZ
8db9f13bc6
Merge branch 'master' into circuit_breaker
2022-06-20 13:42:18 +08:00
geekboood
fdacd94af5
fix: invalid chunk data when failed to read manifests
2022-06-19 16:54:04 +08:00
石昌林
78b3728169
add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations
...
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply
# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply
# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply
# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply
# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
2022-06-15 21:07:55 +08:00
chrislu
ebd6097e6f
fix: sequence read rate is slow when use filer.ChunkReadAt.ReadAt #3074
...
related to https://github.com/chrislusf/seaweedfs/pull/3074
2022-06-07 22:33:35 -07:00
chrislu
4fd5f96598
filer: remove replication, collection, disk_type info from entry metadata
...
these metadata can change and are not used
2022-06-06 00:39:35 -07:00
creeew
02ae102731
fix filer.sync missing source srv uploaded files to target when target down
2022-06-02 01:28:47 +08:00
chrislu
6adc42147f
fresh filer store bootstrap from the oldest peer
2022-05-30 21:27:48 -07:00
chrislu
c59068d0f3
refactor
2022-05-30 16:28:36 -07:00
chrislu
682382648e
collect cluster node start time
2022-05-30 16:23:52 -07:00
chrislu
aece35a64f
stop when on disk log is done
2022-05-30 15:20:51 -07:00
chrislu
a2b101a737
subscribe metadata between a range
2022-05-30 15:04:19 -07:00
chrislu
9219664505
reset sync offset if peer filer resets
...
fix https://github.com/chrislusf/seaweedfs/issues/3089
2022-05-29 13:11:24 -07:00
Kaiwalya Joshi
4598581741
fix: Minor spelling fixes on FilerStorePathTranslator.
2022-05-25 01:11:02 -07:00
chrislu
a5cdcc7e63
fix reading files larger than 20GB due to manifest resolving bug
2022-05-23 01:16:10 -07:00
chrislu
59cc3c0bfd
fix reading files larger than 20GB due to manifest resolving bug
...
this is introduced since Mar 08 2022
2022-05-23 01:14:56 -07:00
Konstantin Lebedev
d12a423aa0
refactor
...
add some tests
https://github.com/chrislusf/seaweedfs/pull/2996
2022-05-12 12:23:02 +05:00
Konstantin Lebedev
f523aed3c9
fixes from comments
...
https://github.com/chrislusf/seaweedfs/pull/2996
2022-05-11 21:50:09 +05:00
chrislu
76da688176
adjust log level
2022-05-11 00:45:19 -07:00
chrislu
a85720f0f6
remove deprecated function
2022-05-10 00:02:23 -07:00
chrislu
791c29ae93
remove deprecated function
...
WithQueryCachePolicyKeepInCache is deprecated and already default behavior
2022-05-10 00:01:43 -07:00
chrislu
10207a48a4
how to setup local test env
2022-05-06 02:25:56 -07:00
chrislu
db33b03735
add comments
2022-05-06 02:19:47 -07:00
chrislu
e350ba8896
larger test range
2022-05-06 02:17:28 -07:00
chrislu
71e0bcc5d5
a bit optimization
2022-05-06 02:11:58 -07:00
chrislu
b2a43dd044
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-06 01:51:28 -07:00
chrislu
d5fe5d12b7
filer: etcd fix pagination
...
fix https://github.com/chrislusf/seaweedfs/issues/3016
2022-05-06 01:51:19 -07:00
Konstantin Lebedev
e6d2cb5911
ydb add empty doc file
2022-05-05 11:12:14 +05:00
Konstantin Lebedev
f58adaab25
build tag ydb since the growth of the binary is 5 mb
2022-05-03 23:20:09 +05:00
Konstantin Lebedev
f127b326bf
add options to scaffold
2022-05-03 22:54:31 +05:00
Konstantin Lebedev
7b544576af
refactor
2022-05-03 20:16:00 +05:00
Konstantin Lebedev
04d5dff6bb
add support native ttl
2022-05-03 17:52:23 +05:00
Konstantin Lebedev
cb3c7a3cdb
enable query cache policy instead of prepare
2022-05-03 16:03:10 +05:00
Konstantin Lebedev
8342f651f3
fix scanNamed
2022-05-03 15:18:28 +05:00
Konstantin Lebedev
d8925b4e83
Merge branch 'new_master' into ydb
...
# Conflicts:
# go.mod
# go.sum
2022-05-03 00:13:57 +05:00
Konstantin Lebedev
7640e650e5
fix queries
2022-05-03 00:11:37 +05:00
Konstantin Lebedev
319d300d48
fix createDB
2022-05-02 22:23:07 +05:00
Konstantin Lebedev
0dc44dda63
ydb do Tx or DB
2022-05-02 15:33:29 +05:00
Konstantin Lebedev
a3e48831f3
ydb BucketAware interface
2022-05-02 12:42:20 +05:00
chrislu
94635e9b5c
filer: add filer group
2022-05-01 21:59:16 -07:00
Konstantin Lebedev
50c4f62ed4
ydb-go-sdk move to v3
2022-05-02 02:07:47 +05:00
Konstantin Lebedev
21033ff4c3
refactor use const CountEntryChunksForGzip
2022-05-01 22:28:55 +05:00
Konstantin Lebedev
ec0ed41e37
ydb kv interface
2022-05-01 21:20:37 +05:00
Konstantin Lebedev
6a052f6ff2
ydb Sql interface
2022-05-01 20:33:03 +05:00
Konstantin Lebedev
275c3bb19c
ydb initial
...
https://github.com/chrislusf/seaweedfs/issues/2942
2022-05-01 17:28:58 +05:00
chrislu
4aa39ef33f
conditionally compile sqlite
2022-04-21 01:22:38 -07:00
chrislu
3885374edf
conditionally build elastic, gocdk to reduce binary size
2022-04-21 01:10:46 -07:00
chrislu
cb1e7aa27d
fix build
2022-04-20 16:21:56 -07:00
a
1d6a9e66b6
Merge branch 'master' into a
2022-04-20 14:01:42 -07:00
justin
3551ca2fcf
enhancement: replace sort.Slice with slices.SortFunc to reduce reflection
2022-04-18 10:35:43 +08:00
eddy-gfx
ec53eec94f
Merge branch 'master' into a
2022-04-06 18:45:55 -05:00
chrislu
bb4beebce3
prevent nil
2022-04-05 10:49:17 -07:00
a
7e92517571
change user and pass to username and password
2022-04-01 14:09:25 -05:00
a
bc603e534f
diff
2022-03-28 16:50:28 +00:00
chrislu
86d4c52d6e
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-26 13:33:20 -07:00
chrislu
21e0898631
refactor: change masters from a slice to a map
2022-03-26 13:33:17 -07:00
Chris Lu
3153c836a7
Merge pull request #2744 from kmlebedev/issues_2545
...
avoid connect to the old filler address
2022-03-26 13:02:49 -07:00
chrislu
cae998eda1
Revert "Revert "remove duplicated metadata subscription in filer""
...
This reverts commit 34b743c481
2022-03-26 13:00:16 -07:00
chrislu
34b743c481
Revert "remove duplicated metadata subscription in filer"
...
This reverts commit 34742be029https://github.com/chrislusf/seaweedfs/issues/2545
2022-03-26 12:33:45 -07:00
chrislu
4e2388e1b5
mount: set file size if it is only on remote gateway
2022-03-21 02:09:54 -07:00
elee
beb406bbbb
fix ls, onBucketDelete still not triggering
2022-03-18 22:21:57 -05:00
elee
411c0df3fe
switch to multi collection, change readme
2022-03-18 21:51:16 -05:00
elee
1cea6c73d3
update readme
2022-03-18 00:34:19 -05:00
elee
25be96832a
update readme with index info
2022-03-18 00:33:08 -05:00
elee
06f23aa675
put in delete folder children query
2022-03-18 00:29:48 -05:00
elee
0701feeb17
added more to readme
2022-03-18 00:26:11 -05:00
elee
171c27ddf1
zz
2022-03-17 22:35:30 -05:00
elee
2f0cdcdceb
update readme
2022-03-17 21:26:16 -05:00
elee
423ce57cde
prefix search, bucket implemented
2022-03-17 21:12:25 -05:00
elee
bf745bdccb
revise comment
2022-03-17 14:09:34 -05:00
elee
a0df993cef
cleanup, add a few comments
2022-03-17 14:08:36 -05:00
elee
94883e2fad
ok then
2022-03-17 05:23:12 -05:00
elee
f3ab6769e9
make start from indexing work
2022-03-17 05:09:13 -05:00
elee
31571dd96e
fix file listing
2022-03-17 05:07:40 -05:00
elee
b25d03340b
Merge branch 'a' of github.com:gfxlabs/seaweedfs into a
2022-03-17 04:51:59 -05:00
elee
921535001a
arangodb adapter
2022-03-17 04:49:26 -05:00
chrislu
6886167d5f
minor optimization
2022-03-17 00:03:28 -07:00
chrislu
4042fdf3bb
rename to skipCheckParentDir
...
related to https://github.com/chrislusf/seaweedfs/pull/2761
It's better to default to false.
2022-03-16 23:55:31 -07:00
zzq09494
40b0033fa7
go fmt
2022-03-17 14:19:48 +08:00
zzq09494
ee5417a5b3
fix: ut test error
2022-03-17 11:16:46 +08:00
zzq09494
81cce4b4c3
filer: support uploading file without needEnsureParentDir
2022-03-17 10:53:47 +08:00
zzq09494
a6a8892255
Revert "filer: support uploading file without needEnsureParentDir"
...
This reverts commit a93c4947ba
2022-03-17 10:27:17 +08:00
zzq09494
a93c4947ba
filer: support uploading file without needEnsureParentDir
2022-03-17 10:18:23 +08:00
chrislu
fbc9f0eb64
minor
2022-03-14 03:19:16 -07:00
chrislu
2f4cd20f90
tests skip reader pattern monitoring
2022-03-13 18:15:53 -07:00
chrislu
53513475bf
mount: add back random read support
...
avoid too much memory used also
2022-03-13 01:38:52 -08:00
Konstantin Lebedev
834210a9dc
avoid connect to the old filler address
...
https://github.com/chrislusf/seaweedfs/issues/2545
2022-03-10 15:24:45 +05:00
zzq09494
2ea18fca48
fix:support some databases which not support 'IF NOT EXISTS' keyword of 'create table command'
2022-03-10 13:37:34 +08:00
banjiaojuhao
f7f2a597dd
minor
2022-03-08 16:22:55 +08:00
banjiaojuhao
d61bea9038
[bugfix] filer: In file modification, old chunks will be mis-deleted when they are merged(Manifestized).
2022-03-08 16:22:55 +08:00
chrislu
6f3ec989a7
fix manifest batch size
2022-03-07 00:07:53 -08:00
banjiaojuhao
a07c93f4dd
filer: reset bytesBuffer before use
2022-03-06 21:07:36 +08:00
chrislu
da76af187f
mount: avoid possible index out of bounds error
2022-03-04 22:36:01 -08:00
chrislu
6fbbc78574
stream reading a whole chunk
2022-03-02 13:50:46 -08:00
chrislu
941ced60a4
download 2 chunks if at the beginning of a file
2022-02-27 03:57:24 -08:00
chrislu
551d00d51a
prefetch other chunks when stream reading
2022-02-26 23:20:45 -08:00
chrislu
7b1a713d2a
remove dead code
2022-02-26 03:23:15 -08:00
chrislu
3a58b7bac6
a little safer
2022-02-26 03:23:06 -08:00
chrislu
3345a50d9b
prefetch 2 chunks
2022-02-26 03:06:17 -08:00
chrislu
86ce69f709
remove logs
2022-02-26 03:00:20 -08:00
chrislu
28b395bef4
better control for reader caching
2022-02-26 02:16:47 -08:00
chrislu
72c0233938
less logs
2022-02-25 15:34:24 -08:00
chrislu
03466f955e
rename: delete source entry metadata only, skipping hard links
2022-02-25 02:57:54 -08:00
chrislu
8080fe4cc1
logs
2022-02-25 02:56:23 -08:00
chrislu
202a29d014
refactoring
2022-02-25 01:17:26 -08:00
chrislu
be3fc77391
mount2: use consistent inode
2022-02-25 00:53:27 -08:00
chrislu
ceaf993a27
mount2: add rdev
2022-02-24 14:51:25 -08:00
chrislu
b05962b90e
rename: handle hard links
2022-02-24 02:59:00 -08:00
chrislu
56da3494cb
leveldb3: adjust memory allocation for each bucket
2022-02-20 13:40:13 -08:00
chrislu
7bc67399e4
listing for filer stores without prefixed query: break if no more progress
2022-02-19 10:12:51 -08:00
chrislu
248c0c8087
minor
2022-02-19 09:54:09 -08:00
chrislu
4ee0a6f47b
filer store: reduce one possible listing operation
2022-02-19 00:43:42 -08:00
chrislu
b3594278c9
optimize a bit
2022-02-18 23:36:10 -08:00
chrislu
df51e0c042
mongodb: remove ErrNoDocuments checking since it is only for creating single record, not for querying
2022-02-15 16:18:36 -08:00
chrislu
98cce6a150
mongodb: ensure closing cursor
2022-02-15 16:15:44 -08:00
banjiaojuhao
b5ec346700
FilerStore: add redis_lua
2022-02-15 20:54:57 +08:00
Eng Zer Jun
b92df1654c
test: use T.TempDir to create temporary test directory
...
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-02-14 10:38:13 +08:00
chrislu
85c1615b43
filer read empty file may cause OOM in some cases
...
fix https://github.com/chrislusf/seaweedfs/issues/2641
2022-02-07 23:08:54 -08:00
chrislu
9405eaefdb
filer.sync: fix replicating partially updated file
...
Run two servers with volumes and fillers:
server -dir=Server1alpha -master.port=11000 -filer -filer.port=11001 -volume.port=11002
server -dir=Server1sigma -master.port=11006 -filer -filer.port=11007 -volume.port=11008
Run Active-Passive filler.sync:
filer.sync -a localhost:11007 -b localhost:11001 -isActivePassive
Upload file to 11007 port:
curl -F file=@/Desktop/9.xml "http://localhost:11007/testFacebook/ "
If we request a file on two servers now, everything will be correct, even if we add data to the file and upload it again:
curl "http://localhost:11007/testFacebook/9.xml "
EQUALS
curl "http://localhost:11001/testFacebook/9.xml "
However, if we change the already existing data in the file (for example, we change the first line in the file, reducing its length), then this file on the second server will not be valid and will not be equivalent to the first file
Снимок экрана 2022-02-07 в 14 21 11
This problem occurs on line 202 in the filer_sink.go file. In particular, this is due to incorrect mapping of chunk names in the DoMinusChunks function. The names of deletedChunks do not match the chunks of existingEntry.Chunks, since the first chunks come from another server and have a different addressing (name) compared to the addressing on the server where the file is being overwritten.
Deleted chunks are not actually deleted on the server to which the file is replicated.
2022-02-07 03:46:28 -08:00
chrislu
4a311c7f5e
dedup local metadata subscribers
...
fix https://github.com/chrislusf/seaweedfs/discussions/2542
2022-01-23 16:14:22 -08:00
chrislu
606667f205
able to configure the quota for a bucket
2022-01-21 01:42:20 -08:00
guol-fnst
da9540e666
add gocql timeout setting
2022-01-18 15:21:13 +08:00
banjiaojuhao
45e9c83421
padding zero for sparse file
2022-01-13 22:21:22 +08:00
chrislu
1453263b63
remove dead code
2022-01-13 02:02:04 -08:00
chrislu
10ecf80ca1
add a debug capability to list all metadata keys
2022-01-11 23:25:04 -08:00
chrislu
34742be029
remove duplicated metadata subscription in filer
...
https://github.com/chrislusf/seaweedfs/issues/2545
2021-12-30 01:51:52 -08:00
chrislu
9a00c17555
reader: avoid wrong pattern detection due to lock waiting
2021-12-28 16:30:33 -08:00
chenkai
47c30e3add
filer list entries use context to break job
2021-12-28 15:03:41 +08:00
chrislu
9f9ef1340c
use streaming mode for long poll grpc calls
...
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
banjiaojuhao
083bf3a137
filer server: add "datacenter, rack and datanode" for path specific configuration
2021-12-23 23:25:05 +08:00
chrislu
b541e39a2c
fix tests
2021-12-22 16:17:30 -08:00
chrislu
0ec7bc6710
detect non streaming mode on the first read
2021-12-22 16:05:38 -08:00
chrislu
4c1368d621
fix test
2021-12-22 16:05:08 -08:00
chrislu
0cb9036f66
mount: only cache the first chunk on stream read
2021-12-19 23:06:03 -08:00
chrislu
a152f17937
mount: improve read performance on random reads
2021-12-19 22:43:14 -08:00
Bl1tz23
83bffca7e6
fix MongodbStore.ListDirectoryEntries panic on find failure
2021-12-03 13:58:37 +03:00
Chris Lu
689f5513a9
redis3 supports sentinel
2021-11-29 01:09:51 -08:00
Chris Lu
2f72c24498
skip the rest logic
2021-11-29 00:32:21 -08:00
Chris Lu
96c66ca2aa
read deleted chunks when replicating data
2021-11-28 23:33:03 -08:00
