mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-22 16:17:55 +08:00
Code Issues Packages Projects Releases Wiki Activity
11,002 Commits 32 Branches 296 Tags 152 MiB
3ef716409e
Commit Graph

1635 Commits

Author SHA1 Message Date
chrislu
b590a68e75 revert to non-streaming mode to assign file id 
fix https://github.com/seaweedfs/seaweedfs/issues/4838#issuecomment-1722290685
 2023-09-16 16:13:08 -07:00
chrislu
4650c4c65f Revert "turn on streaming assign file id" 
This reverts commit 733db2bc88.
 2023-09-16 16:09:20 -07:00
chrislu
89a1fd1751 Squashed commit of the following: 
commit 4827425146
Author: chrislu <chris.lu@gmail.com>
Date:   Sat Sep 16 15:05:38 2023 -0700

    balancer works

commit 3b50139f68
Author: chrislu <chris.lu@gmail.com>
Date:   Fri Sep 15 22:22:32 2023 -0700

    comments

commit 7f685ce7ba
Author: chrislu <chris.lu@gmail.com>
Date:   Fri Sep 15 22:20:05 2023 -0700

    adjust APIs

commit 436d99443b
Author: chrislu <chris.lu@gmail.com>
Date:   Thu Sep 14 23:49:05 2023 -0700

    receive broker stats

commit b771fefa37
Merge: 0a851ec00 890881037
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Sep 13 00:03:47 2023 -0700

    Merge branch 'master' into sub

commit 0a851ec00b
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Sep 10 22:01:25 2023 -0700

    Create balancer.go

commit 39941edc0b
Author: chrislu <chris.lu@gmail.com>
Date:   Thu Sep 7 23:55:19 2023 -0700

    add publisher shutdown

commit 875f562779
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Sep 6 23:16:41 2023 -0700

    server side send response at least once per second

commit 984b6c54cf
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Sep 6 23:15:29 2023 -0700

    ack interval 128

commit 2492a45499
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Sep 6 22:39:46 2023 -0700

    ack interval

commit ba67e6ca29
Author: chrislu <chris.lu@gmail.com>
Date:   Mon Sep 4 21:43:50 2023 -0700

    api for sub

commit 9e4f985698
Author: chrislu <chris.lu@gmail.com>
Date:   Mon Sep 4 21:43:30 2023 -0700

    publish, benchmark

commit cb470d44df
Author: chrislu <chris.lu@gmail.com>
Date:   Fri Sep 1 00:36:51 2023 -0700

    can pub and sub

commit 1eb2da46d5
Author: chrislu <chris.lu@gmail.com>
Date:   Mon Aug 28 09:02:12 2023 -0700

    connect and publish

commit 504ae8383a
Author: chrislu <chris.lu@gmail.com>
Date:   Mon Aug 28 09:01:25 2023 -0700

    protoc version

commit dbcba75271
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 27 18:59:04 2023 -0700

    rename to lookup

commit c9caf33119
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 27 18:33:46 2023 -0700

    move functions

commit 4d6c18d86f
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 27 17:50:59 2023 -0700

    pub sub initial tests

commit 4eb8e8624d
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 27 13:14:39 2023 -0700

    rename

commit 1990456670
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 27 13:13:14 2023 -0700

    sub

commit 905911853d
Author: chrislu <chris.lu@gmail.com>
Date:   Sat Aug 26 13:39:21 2023 -0700

    adjust proto
 2023-09-16 15:06:16 -07:00
chrislu
4827425146 balancer works 2023-09-16 15:05:38 -07:00
Nico D'Cotta
796b7508f3
Implement SRV lookups for filer (#4767) 2023-08-24 07:08:56 -07:00
chrislu
733db2bc88 turn on streaming assign file id 2023-08-23 00:34:21 -07:00
chrislu
99f037b958 streaming assign file ids 2023-08-23 00:31:33 -07:00
chrislu
94b7e2a37c add stream assign server side implementation 2023-08-22 09:59:04 -07:00
chrislu
ccedad5196 refactor files 2023-08-22 09:54:06 -07:00
wusong
33b9192536
Fix large file chunks deleted by mistake (#4678) 2023-07-17 07:52:14 -07:00
Konstantin Lebedev
2c076ebd96
avoid logging nil error (#4659) 
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-07-10 23:38:52 -07:00
chrislu
a315490f7d proxy to master uses http address 
fix https://github.com/seaweedfs/seaweedfs/issues/4607
 2023-07-04 11:45:21 -07:00
chrislu
f193942adc lock on append metadata operation 2023-06-25 22:07:23 -07:00
chrislu
61c42f9991 adjust lock APIs 2023-06-25 20:30:20 -07:00
chrislu
5149b3d07b filer can proxy to peer filer holding the lock 2023-06-25 15:28:01 -07:00
chrislu
868f7875d7 refactor 2023-06-25 14:30:58 -07:00
chrislu
464a71a373 add distributed lock manager 2023-06-25 14:14:40 -07:00
chrislu
3fd659df2a add distributed lock manager 2023-06-25 00:58:21 -07:00
chrislu
8ec1bc2c99 remove unused cluster node leader 2023-06-19 18:19:13 -07:00
chrislu
94fbf02ea2 generate jwt token when looking up needle id 
fix https://github.com/seaweedfs/seaweedfs/issues/4577
 2023-06-17 00:11:23 -07:00
Konstantin Lebedev
25535e9c36
Delete volume is empty (#4561) 
* use onlyEmpty for deleteVolume
https://github.com/seaweedfs/seaweedfs/issues/4559

* fix IsEmpty

* fix test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-12 10:42:44 -07:00
柏杰
68feffa4e6
avoid overwriting non-nil uploadReaderToChunks.uploadErr with nil value (#4560) 2023-06-10 23:15:27 -07:00
Konstantin Lebedev
5ee04d20fa
Healthz check for deadlocks (#4558) 2023-06-09 09:42:48 -07:00
chrislu
0bffb3770f support TTL when creating folders 
related to https://github.com/seaweedfs/seaweedfs/discussions/4549
 2023-06-05 23:29:25 -07:00
wusong
26f15d0079
Fix no more writable volumes by delay judgment (#4548) 
* fix nomore writables volumes while disk free space is sufficient by time delay

* reset

---------

Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-06-05 10:17:21 -07:00
wusong
5aec6da8a3
fix deadlock for filer upload (#4527) 
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-06-03 13:38:27 -07:00
wusong
e7f4418855
add return for error handle (#4502) 2023-05-24 07:45:06 -07:00
chrislu
d999f1f0e2 update raft version 
fix #4460
 2023-05-09 22:54:23 -07:00
chrislu
4511edc871 update raft 2023-05-07 13:33:44 -07:00
zemul
e9fda774f4
[Filer] post add param:saveInside (#4434) 
* fix:mount deadlock

* feat: filer http upload to metadata

* feat: /etc save inside

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2023-04-29 08:31:05 -07:00
wusong
4867aa03ab
Fix: http rename move dir to subdir (#4432) 
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-04-26 20:54:36 -07:00
zemul
f352616b7d
Volume range read use sync.pool (#4422) 2023-04-24 07:00:48 -07:00
wusong
a408b46d95
compilation fail (#4414) 
Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
 2023-04-18 00:36:55 -07:00
chrislu
0a22eea55d collect ec shard from multiple locations 
fix https://github.com/seaweedfs/seaweedfs/issues/4365
 2023-04-17 22:56:21 -07:00
chrislu
ae9388723f adjust error message 2023-04-17 22:01:39 -07:00
Konstantin Lebedev
130bc3e668
s3 fix get fake dir object key (#4390) 2023-04-11 07:36:22 -07:00
wusong
2b82d0f871
style: clerical error (#4361) 2023-03-31 07:36:21 -07:00
Konstantin Lebedev
de4545c28b
add validate config for raft (#4332) 2023-03-21 06:36:13 -07:00
chrislu
adb90bd252 avoid lower casing the command 
fix https://github.com/seaweedfs/seaweedfs/pull/4321
 2023-03-19 21:20:46 -07:00
chrislu
784daea1fa fix volume not found if marked as read only 
fix https://github.com/seaweedfs/seaweedfs/issues/4088
 2023-03-19 18:30:13 -07:00
LHHDZ
d695119073
The filer does not set defaultReplication to the defaultReplication o… (#4315) 
The filer does not set defaultReplication to the defaultReplication of the master (it is not necessary, because if the filer is not set, the configuration of the master will be used when calling Assign), otherwise, when the defaultReplication of the master is modified and all master nodes are restarted, the defaultReplication will not take effect (because the filer Keep the previous defaultReplication setting from the master)

Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2023-03-15 22:10:39 -07:00
Rohit Chormale
59706c89fb
#4270 set http status code to 409 if dir already exists (#4287) 2023-03-08 07:45:08 -08:00
zemul
0bf56298d5
fix chunk.ModifiedTsNs (#4264) 
* fix

* fix mtime s > ns

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2023-03-02 08:24:36 -08:00
chrislu
8241c9a829 adjust ui pagination size 
fix https://github.com/seaweedfs/seaweedfs/issues/4258
 2023-02-28 08:31:26 -08:00
lfhy
1976ca9160
add -disk to filer command (#4247) 
* add -disk to filer command

* add diskType to filer.grpc

* use filer.disk when filerWebDavOptions.disk is empty

* add filer.disk to weed server command.

---------

Co-authored-by: 三千院羽 <3000y@MacBook-Pro.lan>
 2023-02-25 09:48:59 -08:00
Zachary Walters
ef2f741823
Updated the deprecated ioutil dependency (#4239) 2023-02-21 19:47:33 -08:00
Neo
d5f77706a8
volume:fix return if Replicate write error (#4188) 2023-02-06 22:34:56 -08:00
chrislu
c47f75eb5b fix null content type caused exception in hdfs client 
fix https://github.com/seaweedfs/seaweedfs/issues/4170
 2023-02-01 02:38:05 -08:00
a
0d085ec175
make directory lister trigger if mime type is httpd/unix-directory (#4169) 2023-01-31 06:54:40 -08:00
chrislu
81fdf3651b grpc connection to filer add sw-client-id header 2023-01-20 01:48:12 -08:00
chrislu
bfe5d910c6 use one readerCache for the whole file 2023-01-16 22:43:02 -08:00
Guo Lei
83e29faa55
use time.NewTicker instead of time.Tick. (#4119) 2023-01-09 06:47:08 -08:00
monchickey
3e2c9ea73d
Add image cropping. (#4117) 2023-01-06 09:28:07 -08:00
chrislu
7bdae5172e batch delete EC needles 
fix https://github.com/seaweedfs/seaweedfs/issues/4107
 2023-01-03 22:05:26 -08:00
Chris Lu
d4566d4aaa
more solid weed mount (#4089) 
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
 2023-01-02 23:20:45 -08:00
Guo Lei
d8cfa1552b
support enable/disable vacuum (#4087) 
* stop vacuum

* suspend/resume vacuum

* remove unused code

* rename

* rename param
 2022-12-28 01:36:44 -08:00
lfhy
038bab3726
add -filer.path to webdav command (#4061) 2022-12-17 20:07:02 -08:00
chrislu
fc6b9e6e0c volume server logs add url info 
debug https://github.com/seaweedfs/seaweedfs/issues/3964
 2022-12-11 13:15:38 -08:00
chrislu
38c63c429b Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-12-10 21:49:09 -08:00
chrislu
e1ca6308cb add chunk etag when downloading from remote storage 
fix https://github.com/seaweedfs/seaweedfs/issues/3987
 2022-12-10 21:49:07 -08:00
wusong
549354e324
Fix hardlink counting (#4042) 
Signed-off-by: wusong <wangwusong@virtaitech.com>

Signed-off-by: wusong <wangwusong@virtaitech.com>
Co-authored-by: wusong <wangwusong@virtaitech.com>
 2022-12-08 10:50:57 -08:00
Jiffs Maverick
5f59d43c88
[filer] Fix uri escape during renaming (#4039) 2022-12-07 07:47:40 -08:00
famosss
15284e2bee
fix: Lost Content-Encoding metadata (#4010) 2022-11-23 22:52:34 -08:00
chrislu
70a4c98b00 refactor filer_pb.Entry and filer.Entry to use GetChunks() 
for later locking on reading chunks
 2022-11-15 06:33:36 -08:00
Guo Lei
5b905fb2b7
Lazy loading (#3958) 
* types packages is imported more than onece

* lazy-loading

* fix bugs

* fix bugs

* fix unit tests

* fix test error

* rename function

* unload ldb after initial startup

* Don't load ldb when starting volume server if ldbtimeout is set.

* remove uncessary unloadldb

* Update weed/command/server.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* Update weed/command/volume.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

Co-authored-by: guol-fnst <goul-fnst@fujitsu.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-11-14 00:19:27 -08:00
chrislu
aae82dca6f file name escape uri 
fix https://github.com/seaweedfs/seaweedfs/issues/3969
 2022-11-13 23:01:56 -08:00
chrislu
ea2637734a refactor filer proto chunk variable from mtime to modified_ts_ns 2022-10-28 12:53:19 -07:00
Eric Yang
51d462f204
ADHOC: volume fsck using append at ns (#3906) 
* ADHOC: volume fsck using append at ns

* nit

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-10-24 22:09:38 -07:00
liubaojiang
25471d579a
add md5 header when UploadData to replication in ReplicatedWrite (#3881) 2022-10-19 01:34:14 -07:00
chrislu
de286fe662 shell: volume.move handles volume moved to cloud tier 
fix https://github.com/seaweedfs/seaweedfs/issues/3803
 2022-10-16 17:52:22 -07:00
Konstantin Lebedev
2f72103c83
avoid load volume file with BytesOffset mismatch (#3841) 
* avoid load volume file with BytesOffset mismatch

https://github.com/seaweedfs/seaweedfs/issues/2966

* set BytesOffset if has not VolumeInfoFile

* typos fail => failed

* exit if bytesOffset mismatch
 2022-10-14 00:18:09 -07:00
Konstantin Lebedev
1f7e52c63e
vacuum metrics and force sync dst files (#3832) 2022-10-13 00:51:20 -07:00
zemul
a05725aea6
filer: get directory metadata (#3833) 2022-10-12 07:14:49 -07:00
LHHDZ
d21e2f523d
split ExtAcpKey to ExtAmzOwnerKey and ExtAmzAclKey to avoid unn… (#3824) 
split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unnecessary `json.Unmarshal()` call

Signed-off-by: changlin.shi <changlin.shi@ly.com>

Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2022-10-11 20:14:14 -07:00
Konstantin Lebedev
5db25a8f2a
metric shows who is currently blocking the cluster or not (#3799) 
* master_admin_lock Shows whether cluster is locked now or not
https://github.com/seaweedfs/seaweedfs/issues/3452

* fix metric MasterAdminLock
 2022-10-07 13:26:29 -07:00
Konstantin Lebedev
a522507f95
configure raft metrics (#3798) 2022-10-07 04:20:34 -07:00
Konstantin Lebedev
b9933d5589
master server graceful stop (#3797) 2022-10-06 09:30:30 -07:00
Konstantin Lebedev
36daa7709d
show raft leader via shell (#3796) 2022-10-06 07:10:41 -07:00
LHHDZ
3de1e19780
s3: sync bucket info from filer (#3759) 2022-09-29 12:29:01 -07:00
Konstantin Lebedev
301b678147
[volume] Add new volumes to HUP(reload) signal (#3755) 
Add new volumes to HUP(reload) signal
 2022-09-28 12:44:13 -07:00
jerebear12
190afcc3fd
Allow CORS on Filer (#3750) 
Adjusted filer response headers
 2022-09-27 11:47:52 -07:00
chrislu
b90d92fca9 grpc: watch metadata changes by directory 2022-09-20 09:25:18 -07:00
chrislu
23f3aea4cf adjust logs 2022-09-18 18:14:24 -07:00
chrislu
10d5b4b32b volume server: rename readBufferSize to readBufferSizeMB 2022-09-17 10:56:28 -07:00
Konstantin Lebedev
5d87ad72d8
mute log filer: no entry is found in filer store (#3707) 2022-09-16 09:05:38 -07:00
famosss
d949a238b8
volume: add "readBufSize" option to customize read optimization (#3702) 
* simplify a bit

* feat: volume: add "readBufSize" option to customize read optimization

* refactor : redbufSIze -> readBufferSize

* simplify a bit

* simplify a bit
 2022-09-16 00:30:40 -07:00
Ryan Russell
c4fc007630
refactor(raft_hashicorp): timeSpeep -> timeSleep (#3694) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 03:12:30 -07:00
Ryan Russell
c202d4c1b0
refactor(master_grpc_server_cluster): seletedSet -> selectedSet (#3693) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 03:12:13 -07:00
chrislu
f9e3e7d1c4 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-15 03:11:34 -07:00
chrislu
896a85d6e4 volume: add "hasSlowRead" option to customize read optimization 2022-09-15 03:11:32 -07:00
Ryan Russell
92658b05a8
refactor(volume_grpc_client_to_master): grpcConection -> `grpcConne… (#3690) 
refactor(volume_grpc_client_to_master): `grpcConection` -> `grpcConnection`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 02:04:29 -07:00
chrislu
21c0587900 go fmt 2022-09-14 23:06:44 -07:00
chrislu
c8645fd232 master: implement grpc VolumeMarkWritable 
fix https://github.com/seaweedfs/seaweedfs/issues/3657
 2022-09-14 23:05:30 -07:00
Ryan Russell
3429cbe3fc
refactor(webdav_server): modifiledTime -> modifiedTime (#3676) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 12:20:27 -07:00
Ryan Russell
d5239ab983
refactor(server): suppported -> supported (#3677) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 12:16:39 -07:00
chrislu
58d18b68d8 rename variable 2022-09-14 00:06:54 -07:00
Konstantin Lebedev
f8ef25099c
Removing chunks on failed to write to replicas (#3591) 
* Removing chunks on failed to write to replicas
https://github.com/seaweedfs/seaweedfs/issues/3578

* put with in the util.Retry

* just purge on any errors
 2022-09-13 22:33:35 -07:00
chrislu
ce55922884 fixing 2022-09-12 09:43:54 -07:00
Konstantin Lebedev
721c6197f9
skip deltaBeat if dn is zero (#3630) 
* skip deltaBeat
https://github.com/seaweedfs/seaweedfs/issues/3629

* fix GrpcPort

* skip url :0

* skip empty DataCenter or Rack

* skip empty heartbeat Ip

* dell msg add DataCenter

* comment todo

* fix
 2022-09-11 22:31:53 -07:00
chrislu
b9112747b5 volume server: synchronously report volume readonly status to master 
fix https://github.com/seaweedfs/seaweedfs/issues/3628
 2022-09-11 19:29:10 -07:00
chrislu
9b084d4c88 purge tcp implementation 2022-09-08 18:03:43 -07:00
Konstantin Lebedev
31d2f77ceb
refactor https://github.com/seaweedfs/seaweedfs/pull/3616 (#3625) 2022-09-07 23:23:33 -07:00
chrislu
7de112943b Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-07 20:14:11 -07:00
chrislu
7e52b1e195 return err for processRangeRequest() 2022-09-07 20:14:09 -07:00
famosss
449582343f
fix:Sometimes a nil pointer exception is thrown (#3618) 2022-09-07 18:57:13 -07:00
famosss
9678fc2106
fix: volume heartbeat processing error (#3616) 2022-09-07 09:48:51 -07:00
famosss
5ff33eb558
fix: volume Deadlock when exception is thrown (#3613) 
* fix: volume Deadlock when exception is thrown

* adjust log level
 2022-09-06 23:56:29 -07:00
Eric Yang
b324a6536c
ADHOC: add read needle meta grpc (#3581) 
* ADHOC: add read needle meta grpc

* add test

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-09-06 23:51:27 -07:00
Konstantin Lebedev
3c3682fcce
more log detail for upload err and deleting (#3577) 2022-09-06 08:19:13 -07:00
Konstantin Lebedev
5307cadb32
[s3] Replace code 500 to 499 for upload with ErrUnexpectedEOF (#3597) 2022-09-05 07:47:43 -07:00
famosss
7f56a40553
Turn down the 404 log level (#3585) 2022-09-04 18:17:01 -07:00
Konstantin Lebedev
90d55cd179
avoid data race read fileChunksSize (#3570) 
https://github.com/seaweedfs/seaweedfs/issues/3557
 2022-09-01 10:31:57 -07:00
Konstantin Lebedev
82c2ce74db
more error details in the log (#3568) 
* more error details in the log
https://github.com/seaweedfs/seaweedfs/issues/3567

* format message
 2022-09-01 09:04:15 -07:00
Konstantin Lebedev
77098de69a
migrate from boltdb to bbolt (#3554) 2022-08-30 10:12:02 -07:00
Konstantin Lebedev
105702ebe0
avoid data race access to uploadReaderToChunks.uploadErr (#3550) 
avoid data race access to uploadErr
https://github.com/seaweedfs/seaweedfs/issues/3549
 2022-08-30 00:03:01 -07:00
Rain Li
089446f5b1
Fix reload page before all file uploaded bug (#3534) 2022-08-28 07:51:47 -07:00
chrislu
3741779144 minor 2022-08-27 00:20:58 -07:00
chrislu
580a7fa685 adjust for data race 
fix https://github.com/seaweedfs/seaweedfs/issues/3512
 2022-08-27 00:19:13 -07:00
famosss
dc4037925d
fix: Build DeletedVids before reset dn's children (#3530) 2022-08-26 22:52:08 -07:00
chrislu
3f3a1341d8 make CodeQL happy 2022-08-26 17:09:11 -07:00
Konstantin Lebedev
e90ab4ac60
avoid race conditions for OnPeerUpdate (#3525) 
https://github.com/seaweedfs/seaweedfs/issues/3524
 2022-08-26 10:18:49 -07:00
Patrick Schmidt
2930263dfd
Fix race conditions during in-flight size checks (#3505) 2022-08-24 20:03:34 -07:00
Patrick Schmidt
7b424a54dc
Add raft server access mutex to avoid races (#3503) 2022-08-24 09:49:05 -07:00
Konstantin Lebedev
f7aeb06544
s3: report metadata if the directory is explicitly created (#3498) 
* replace mkdir to mkFile

* ContentLength must be zero

* revert mkDir

* Seaweedfs-Is-Directory-Key return metadata
 2022-08-24 00:15:44 -07:00
chrislu
10414fd81c ping timeout at 15 seconds 
this 72 minute timeout setting seems unreasonably long

15 seconds is around the time when a new raft leader should be elected.
 2022-08-23 23:28:16 -07:00
askeipx
2e78a522ab
remove old raft servers if they don't answer to pings for too long (#3398) 
* remove old raft servers if they don't answer to pings for too long

add ping durations as options

rename ping fields

fix some todos

get masters through masterclient

raft remove server from leader

use raft servers to ping them

CheckMastersAlive for hashicorp raft only

* prepare blocking ping

* pass waitForReady as param

* pass waitForReady through all functions

* waitForReady works

* refactor

* remove unneeded params

* rollback unneeded changes

* fix
 2022-08-23 23:18:21 -07:00
chrislu
9fce75607d s3: report http.StatusOK if the directory is explicitly created 
fix https://github.com/seaweedfs/seaweedfs/issues/3457
 2022-08-23 01:16:46 -07:00
chrislu
601ba5fb68 master: disable locking if not leader 2022-08-22 14:11:48 -07:00
chrislu
f4b52d4c52 fix format 2022-08-22 00:20:21 -07:00
qzh
74b53729e1
feat(weed.move): add a speed limit parameter of moving files (#3478) 
* feat(weed.move): add a speed limit parameter of moving files

* fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond

Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
 2022-08-21 23:08:31 -07:00
chrislu
fef9c6a520 filer: do not always save files in "/etc" folder to filer store 
fix https://github.com/seaweedfs/seaweedfs/issues/3476
 2022-08-21 17:02:16 -07:00
chrislu
2762154130 fix compilation 2022-08-21 16:09:23 -07:00
chrislu
fb5808e0c3 EC: with multiple volume locations, the ec rebuilding may fail 2022-08-21 15:04:50 -07:00
chrislu
5790d01c6f EC: after ec encoding, the source ec shards may fail to purge if the volume server has multiple disk locations 
related to https://github.com/seaweedfs/seaweedfs/issues/3459
 2022-08-21 14:52:37 -07:00
chrislu
65ff7198fe adjust type 2022-08-20 22:37:24 -07:00
chrislu
28b862f45f use util.Retry to retry 2022-08-20 22:03:27 -07:00
chrislu
3bf8e772f8 webdav: retryable data chunk upload 2022-08-20 19:18:12 -07:00
chrislu
f8fa430257 filer: retryable data chunk upload 2022-08-20 19:15:44 -07:00
chrislu
689b4ecdcc remove unused collection and replication from upload result 2022-08-20 18:14:57 -07:00
chrislu
fdd8c5d5e0 Filer if-modified-since check doesn't fire on "exact" #3467 
fix https://github.com/seaweedfs/seaweedfs/issues/3467
 2022-08-19 22:21:51 -07:00
chrislu
aac45f3e89 filer: retryable when error is not found 2022-08-19 03:36:15 -07:00
Eric Yang
b6f99c4d83
fix naming typo (#3465) 
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-08-18 23:25:32 -07:00
chrislu
eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
chrislu
e259052a70 volume: fix metrics for volume request time 2022-08-17 04:29:31 -07:00
chrislu
789d9dbfe1 webdav: fix nil 
fix https://github.com/seaweedfs/seaweedfs/issues/3440
 2022-08-14 23:33:30 -07:00
Konstantin Lebedev
4d4cd0948d
avoid infinite loop WaitUntilConnected() (#3431) 
https://github.com/seaweedfs/seaweedfs/issues/3421
 2022-08-11 15:03:26 -07:00
Eric Yang
986daecbfe
add upload response (#3428) 
* add upload response

* fixing comments

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-08-10 22:43:09 -07:00
chrislu
ae93c966d9 ensure memory is aligned 
fix https://github.com/seaweedfs/seaweedfs/issues/3427
 2022-08-10 22:27:13 -07:00
LHHDZ
84ec68e11a
Add download speed limit support (#3408) 2022-08-05 01:16:42 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center (#3405) 
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
 2022-08-04 17:35:00 -07:00
Konstantin Lebedev
22181dd018
refactor FilerRequest metrics (#3402) 
* refactor FilerRequest metrics

* avoid double count proxy

* defer to
 2022-08-04 01:44:54 -07:00
Chris Lu
fc8035d672
Merge pull request #3394 from kmlebedev/metricsReplicatedWrite 
Detailed metrics VolumeServerRequestHistogram for writing to disk and replication
 2022-08-01 13:31:01 -07:00
Konstantin Lebedev
df9cc31a0a refactor 2022-08-02 01:18:26 +05:00
Konstantin Lebedev
fc65122766 rename to LoadAvg_1M 2022-08-01 21:32:21 +05:00
Konstantin Lebedev
2f85e31f5c Merge branch 'master' into gentle_vacuum 2022-08-01 20:51:58 +05:00
Konstantin Lebedev
5209ebbeef remove percent 2022-08-01 20:40:38 +05:00
Chris Lu
b59bc607bf
Merge pull request #3338 from kmlebedev/issues/3083 
rollback over onPeerUpdate implementation of automatic clean-up of failed servers in favor of synchronous ping
 2022-08-01 08:23:10 -07:00
Konstantin Lebedev
cd5c7ad052 move to github.com/seaweedfs/seaweedfs 2022-08-01 16:36:32 +05:00
Konstantin Lebedev
3c75479e2b Merge branch 'master' into gentle_vacuum 
# Conflicts:
#	weed/pb/messaging_pb/messaging.pb.go
#	weed/pb/messaging_pb/messaging_grpc.pb.go
#	weed/pb/s3_pb/s3.pb.go
#	weed/pb/volume_server_pb/volume_server.pb.go
#	weed/server/volume_grpc_vacuum.go
 2022-08-01 14:45:22 +05:00
Konstantin Lebedev
1d29f67c02 revert disk stats 2022-08-01 14:29:41 +05:00
Konstantin Lebedev
a98f6d66a3 rollback over onPeerupdate implementation of automatic clean-up of failed servers in favor of synchronous ping 2022-08-01 12:51:41 +05:00
Patrick Schmidt
1a4a36d510
Add healthy indicator for raft status 2022-07-30 19:34:26 +02:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Konstantin Lebedev
2f0dda384d vacuum show LA 2022-07-29 11:59:33 +05:00
chrislu
9f479aab98 allocate brokers to serve segments 2022-07-28 23:24:38 -07:00
chrislu
bb01b68fa0 refactor 2022-07-28 23:24:38 -07:00
chrislu
8060fdcac5 remove old code 2022-07-28 23:24:38 -07:00
chrislu
f25e273e32 display data center and rack in cluster.ps 2022-07-28 23:22:52 -07:00
chrislu
68065128b8 add dc and rack 2022-07-28 23:22:51 -07:00
chrislu
3828b8ce87 "github.com/chrislusf/raft" => "github.com/seaweedfs/raft" 2022-07-27 12:12:40 -07:00
Konstantin Lebedev
c88ea31f62 fix RUnlock of unlocked RWMutex 2022-07-26 12:57:07 +05:00
chrislu
64f3d6fb6e metadata subscription uses client epoch 2022-07-23 10:50:28 -07:00
ningfd
58f2dd6740 change topology mapping definition, improve spelling 2022-07-22 17:18:56 +08:00
Konstantin Lebedev
3c42814b58 avoid deadlock 2022-07-21 17:15:10 +05:00
Konstantin Lebedev
93ca87b7cb use safe onPeerUpdateDoneCns 2022-07-21 15:51:14 +05:00
Konstantin Lebedev
7875470e74 onPeerUpdateGoroutineCount use int32 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
6c390851e7 fix design 2022-07-20 18:08:12 +05:00
Konstantin Lebedev
f6a966b4fc add waiting log message 2022-07-20 00:31:57 +05:00
Konstantin Lebedev
6cfbfb0849 check for ping before deleting raft server 
https://github.com/chrislusf/seaweedfs/issues/3083
 2022-07-20 00:04:12 +05:00
Chris Lu
f0d560060f
Merge pull request #3336 from kmlebedev/raftSrvRemove 
raftServerRemoval typo and logs
 2022-07-19 11:32:17 -07:00
Konstantin Lebedev
f419d5643a fix typo 
add remove logs
 2022-07-19 11:50:52 +05:00
chrislu
aca20cd9f4 more logging related to filer metadata subscription 2022-07-15 00:42:13 -07:00
chrislu
fbd8f868a1 filer may have trouble to re-connect 
clientId is used twice: one for local metadata subscription, one for combined metadata subscription.
 2022-07-14 12:15:31 -07:00
liubaojiang
4e83e92156 add subscriber clientId if it is the first time 2022-07-08 20:27:25 +08:00
xdadrm
3a42269da8
Fixes WebDAV 0-bytes files 
Fixes the issue where files created via WebDAV show as 0-bytes size when read via fuse.
 2022-06-27 20:19:21 +02:00
Konstantin Lebedev
6c20a3b622 avoid set currentMaster k8s svc.local discoveruy service domains 
https://github.com/chrislusf/seaweedfs/issues/2589
 2022-06-27 21:47:05 +05:00
chrislu
9dfea44aeb fix unit 2022-06-26 12:21:38 -07:00
chrislu
3142e77eca mount: stats report physical size instead of logical size 2022-06-26 12:14:34 -07:00
chrislu
0c13fc873c fix compilation 2022-06-23 00:45:11 -07:00
chrislu
96496d5286 master: broadcast new volume locations to clients to avoid possible racing condition 
fix https://github.com/chrislusf/seaweedfs/issues/3220
 2022-06-23 00:41:33 -07:00
Konstantin Lebedev
c1c8dad677 avoid no such raft date directory 
https://github.com/chrislusf/seaweedfs/issues/3214
 2022-06-21 13:47:51 +05:00
石昌林
81f7f08708 Determine whether to preallocate according to the master configuration before executing copy volume 2022-06-20 21:12:44 +08:00
chrislu
aaa9938353 Revert "adjust conditions" 
This reverts commit e024586ff1.
 2022-06-16 01:26:36 -07:00
chrislu
e024586ff1 adjust conditions 2022-06-15 23:19:39 -07:00
Chris Lu
82f3bcc65e
Merge pull request #3081 from paochiang/volume_upload_limit_fix 
exclude replication from the concurrentUploadLimitMB
 2022-06-15 23:16:44 -07:00
liubaojiang
df0ce31a2e add condition when inFlightUploadDataLimitCond signal 2022-06-16 14:07:11 +08:00
liubaojiang
3076ac101e move vs.concurrentUploadLimit != 0 out of the lock 2022-06-16 09:58:44 +08:00
Chris Lu
55b9f281d1
Merge pull request #3185 from kmlebedev/logging_err_process_range_req 2022-06-15 06:33:33 -07:00
Konstantin Lebedev
31a929f7f2 logging processRangeRequest errors 2022-06-15 15:36:43 +05:00
Chris Lu
4eeeb5f50d
Merge pull request #3180 from blacktear23/issue-3149 2022-06-15 00:55:20 -07:00