chrislu
40089b6753
maintain node lookup counter
2022-07-24 13:43:11 -07:00
chrislu
d24db396cc
fix removing path from inode2path
2022-07-23 18:20:29 -07:00
chrislu
41eeb4deef
do not add new inode during link
2022-07-23 13:37:07 -07:00
chrislu
02c2d81cde
fix removePathFromInode2Path
2022-07-23 13:21:42 -07:00
chrislu
4d5dc55766
mount: better support hard links
2022-07-23 12:57:00 -07:00
chrislu
ebda0689ab
use linux default max name length
2022-07-23 10:51:06 -07:00
chrislu
64f3d6fb6e
metadata subscription uses client epoch
2022-07-23 10:50:28 -07:00
chrislu
2c8818351f
mount: comment out stuck fuse inode notify during renaming
2022-07-22 18:17:34 -07:00
chrislu
a5c94acd4f
mount: check possible nil entry case
2022-07-22 15:33:37 -07:00
Chris Lu
0716092b39
Merge pull request #3350 from shichanglin5/optimize_masterclient_vidmap
...
Solve the problem that `LookupFileId` lookup urls is empty due to lea…
2022-07-22 02:57:50 -07:00
LHHDZ
994a2dec78
fix vid_map_test tests
2022-07-22 17:54:57 +08:00
LHHDZ
58c02d6429
Solve the problem that LookupFileId lookup urls is empty due to leader switching
...
The vidMap structure is modified to a linked list structure (the length is limited to 5). When the vidMap is reset, the current vidMap is added to the new vidMap as a cache node. When the query locations is empty, the cache node is searched to avoid problems when the master switches leaders.
2022-07-22 17:22:38 +08:00
ningfd
58f2dd6740
change topology mapping definition, improve spelling
2022-07-22 17:18:56 +08:00
ningfd
6f882eb354
optimiz: master ui will render data in order
2022-07-22 14:37:49 +08:00
chrislu
7a6c559ab4
fix Change replication via volume.configure.replication by collection
...
fix https://github.com/chrislusf/seaweedfs/issues/3346
2022-07-21 22:01:05 -07:00
chrislu
fc8241fb5e
leveldb3: add instant dropping bucket
2022-07-21 18:48:51 -07:00
chrislu
c93f7ffa44
explicit bucket aware declaration
2022-07-21 18:23:53 -07:00
Konstantin Lebedev
3c42814b58
avoid deadlock
2022-07-21 17:15:10 +05:00
Konstantin Lebedev
93ca87b7cb
use safe onPeerUpdateDoneCns
2022-07-21 15:51:14 +05:00
chrislu
6147b61b0a
mount: fix concurrent map read and map write
...
fix https://github.com/chrislusf/seaweedfs/issues/3344
2022-07-21 01:38:26 -07:00
Chris Lu
9ec0d1caaa
Merge pull request #3337 from guol-fnst/loading_volume
2022-07-20 09:06:06 -07:00
Konstantin Lebedev
7875470e74
onPeerUpdateGoroutineCount use int32
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
c5189c343b
remove ticker
...
update the topology before each file
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
e2d991d8d0
ticker.Stop
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
de4fcc0e2c
sync update topologyInfo
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
fa88dff7ce
update otherNodes
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
72dca31cfa
fix update topologyInfo
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
884ffbafee
clouse background update
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
39eaf426f8
fix TestVolumeServerEvacuate
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
b5e5f6f55a
update topologyInfo
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
867269cdcf
help rack
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
6f764e1014
volume server evacuate from rack
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
ba0e3ce5fa
volume server evacuate to target server
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
b6471ecd75
err msg with duplicated local subscription detected move to log level 1
...
https://github.com/chrislusf/seaweedfs/issues/3320
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
6c390851e7
fix design
2022-07-20 18:08:12 +05:00
Tuan Vuong
312e13416b
filter duplicated action
2022-07-20 16:43:50 +07:00
guol-fnst
ac694f0c8f
rename parameter and reuse functions
...
rename milestone to watermark
2022-07-20 17:00:40 +08:00
Chris Lu
4820b594a7
Merge pull request #3298 from kmlebedev/evacuateTargetServer
...
volume server evacuate to target server
2022-07-19 13:56:49 -07:00
Konstantin Lebedev
d3f7c09c03
remove ticker
...
update the topology before each file
2022-07-20 00:54:23 +05:00
Konstantin Lebedev
11e393dbe7
err msg with duplicated local subscription detected move to log level 1
...
https://github.com/chrislusf/seaweedfs/issues/3320
2022-07-20 00:45:13 +05:00
Konstantin Lebedev
f6a966b4fc
add waiting log message
2022-07-20 00:31:57 +05:00
Konstantin Lebedev
6cfbfb0849
check for ping before deleting raft server
...
https://github.com/chrislusf/seaweedfs/issues/3083
2022-07-20 00:04:12 +05:00
chrislu
2ae3f812f8
minor
2022-07-19 11:43:31 -07:00
Chris Lu
9667588af0
Merge pull request #3318 from kmlebedev/issues/3310
...
Use fallback if urls are not found
2022-07-19 11:38:43 -07:00
Chris Lu
f0d560060f
Merge pull request #3336 from kmlebedev/raftSrvRemove
...
raftServerRemoval typo and logs
2022-07-19 11:32:17 -07:00
guol-fnst
91285bb51d
remove uncessary code
2022-07-19 20:00:44 +08:00
guol-fnst
b9256e0b34
optimiz
2022-07-19 16:58:53 +08:00
Konstantin Lebedev
f419d5643a
fix typo
...
add remove logs
2022-07-19 11:50:52 +05:00
guol-fnst
bec9f79659
use snapshot
2022-07-19 11:14:41 +08:00
Konstantin Lebedev
d422e7769c
ticker.Stop
2022-07-18 16:38:19 +05:00
Konstantin Lebedev
73a0dea16b
sync update topologyInfo
2022-07-18 16:27:02 +05:00
chrislu
56ec89625a
3.16
2022-07-18 00:53:05 -07:00
Konstantin Lebedev
2b4112e462
update otherNodes
2022-07-18 11:32:28 +05:00
Konstantin Lebedev
3c2774ec3d
fix update topologyInfo
2022-07-18 01:46:31 +05:00
Konstantin Lebedev
7b1497ee63
Use BackoffSchedule for getLookupFileId
2022-07-15 16:05:35 +05:00
Konstantin Lebedev
01996bccf8
Use fallback if urls are not found
2022-07-15 15:29:15 +05:00
Konstantin Lebedev
4d5144e50d
clouse background update
2022-07-15 13:51:08 +05:00
chrislu
1db012485f
mount: fix bug during busy writes
...
fix https://github.com/chrislusf/seaweedfs/issues/3315
2022-07-15 01:03:17 -07:00
chrislu
aca20cd9f4
more logging related to filer metadata subscription
2022-07-15 00:42:13 -07:00
Chris Lu
abeb7f65f8
Merge pull request #3314 from shichanglin5/fix_ETag
...
File upload should succeed should return `ETag` instead of `Etag`
2022-07-14 15:35:41 -07:00
chrislu
fa61074513
add clientId logging
2022-07-14 12:27:34 -07:00
chrislu
fbd8f868a1
filer may have trouble to re-connect
...
clientId is used twice: one for local metadata subscription, one for combined metadata subscription.
2022-07-14 12:15:31 -07:00
LHHDZ
c08f6d0ff7
File upload should succeed should return ETag instead of Etag
...
at `/go-1.18.1/src/net/textproto/header.go:39`, header is canonicalized by `textproto.CanonicalMIMEHeaderKey`
2022-07-14 19:45:01 +08:00
Chris Lu
957c9a2eb2
Merge pull request #3309 from guol-fnst/loading_volume
...
optimiz concurrency
2022-07-13 23:12:24 -07:00
guol-fnst
300b383cdf
use 10 or numCPU workers if env is not found
2022-07-14 14:06:46 +08:00
guol-fnst
308a48c0c2
optimiz concurrency
...
user can customize number of workers via env "GOMAXPROCS"
2022-07-14 09:57:25 +08:00
Chris Lu
1a130125de
Merge pull request #3308 from guo-sj/check_duplicate_ak
2022-07-13 09:23:37 -07:00
guosj
354b7bdff0
replace errors.New(fmt.Sprintf(...)) with fmt.Errorf()
2022-07-13 22:49:03 +08:00
chrislu
289402a741
writer pattern: similar changes to reader pattern
2022-07-13 02:30:44 -07:00
guosj
ab1b9697e6
supplement check duplicate accesskey
2022-07-13 17:28:20 +08:00
chrislu
6e90f7bdd0
detect sequential mode with a counter
2022-07-13 02:20:03 -07:00
chrislu
578c8aea0e
typo
2022-07-13 02:18:31 -07:00
chrislu
1451b389a4
Avoid fatal error: sync: Unlock of unlocked RWMutex
...
fix https://github.com/chrislusf/seaweedfs/issues/3306
2022-07-13 00:58:15 -07:00
石昌林
392da4e038
Make the prompt information clearer
2022-07-13 11:39:45 +08:00
石昌林
ad5e8f68ec
Check whether there is a duplicate accessKey when modifying iam
2022-07-13 11:36:11 +08:00
Chris Lu
fba5219dab
Merge pull request #3303 from guol-fnst/loading_volume
...
speeding up loading volumes
2022-07-12 03:01:24 -07:00
Konstantin Lebedev
6622240df7
fix TestVolumeServerEvacuate
2022-07-12 14:56:34 +05:00
chrislu
860c207c94
typo
2022-07-12 02:56:19 -07:00
Konstantin Lebedev
8372721a62
update topologyInfo
2022-07-12 13:47:21 +05:00
guol-fnst
26313060a3
speeding up loading volumes
2022-07-12 16:19:33 +08:00
Konstantin Lebedev
ee95d23a22
help rack
2022-07-12 11:56:58 +05:00
Konstantin Lebedev
087fa1347f
volume server evacuate from rack
2022-07-12 11:33:08 +05:00
Chris Lu
de97ba94c3
Merge pull request #3302 from guo-sj/fix_put_user_policy
2022-07-11 21:15:37 -07:00
guosj
b3076909d1
refactor code
2022-07-12 11:25:36 +08:00
Konstantin Lebedev
4236c36599
volume server evacuate to target server
2022-07-11 16:58:15 +05:00
Chris Lu
7929a50327
Merge pull request #3296 from guo-sj/fix_put_user_policy
...
refine PutUserPolicy
2022-07-11 01:49:25 -07:00
guosj
7dbf19d09a
refine PutUserPolicy
2022-07-11 16:38:41 +08:00
chrislu
37578929d4
fix compilation
2022-07-11 01:30:00 -07:00
chrislu
93ccc6e05f
3.15
2022-07-11 00:50:07 -07:00
chrislu
f87952a411
remove locks
2022-07-11 00:25:20 -07:00
guosj
5469efa336
close SkipCheckParentDirectory flag
2022-07-11 09:44:48 +08:00
chrislu
3d0defa663
fix compilation
2022-07-08 19:16:58 -07:00
chrislu
6f00c7f061
comments
2022-07-08 19:07:31 -07:00
chrislu
d685b94102
mount: rename also invalidate source inode
2022-07-08 19:07:03 -07:00
chrislu
48382676d2
fix filtering by volume id
2022-07-08 10:29:24 -07:00
liubaojiang
4e83e92156
add subscriber clientId if it is the first time
2022-07-08 20:27:25 +08:00
guosj
5b3192344d
fix iam CreatePolicy error
2022-07-08 17:42:21 +08:00
chrislu
e2ecf137a2
fix resetting fh.reader
2022-07-08 01:04:15 -07:00
chrislu
185332b91f
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-07-08 00:29:41 -07:00
chrislu
28add5a534
mount: fix racing conditions
...
prevent wrong reading when the SingleChunkCacher is started, but not finished yet
2022-07-08 00:29:39 -07:00
duanhongyi
1ceab96aba
filer tikv support tls
2022-07-08 14:23:06 +08:00
chrislu
a85ed3fe8f
minor
2022-07-07 22:27:24 -07:00
chrislu
2ea18cdcc8
remove dead code
2022-07-07 22:26:03 -07:00
chrislu
b7de5c6c43
shell: remove unused filer parameter from shell.toml
2022-07-07 15:01:23 -07:00
chrislu
f2f0482dd3
mount: random read also try to use the local cache first
2022-07-07 11:50:28 -07:00
chrislu
05c53820b9
mount: file handler release memory
2022-07-07 11:49:40 -07:00
chrislu
69ef6459c3
S3 ListObjectsV2 fix pagination under bucket root
...
fix https://github.com/chrislusf/seaweedfs/issues/3166
2022-07-06 02:28:34 -07:00
Konstantin Lebedev
785223e587
rabbitpubsub enable durable
2022-07-06 10:05:29 +05:00
chrislu
57975b93ff
adjust used size reporting
2022-07-04 13:47:42 -07:00
chrislu
a79f5bd0d5
adjust used size reporting
2022-07-04 13:46:32 -07:00
chrislu
3c79c77056
3.14
2022-07-04 13:15:49 -07:00
guol-fnst
5bc895673e
add getting bucket quota
2022-07-04 15:36:44 +08:00
guol-fnst
dd4664e610
fix usage
2022-07-04 11:31:39 +08:00
chrislu
141f662734
edge case checking when volume server does not have capacity to balance
...
fix https://github.com/chrislusf/seaweedfs/issues/3257
2022-07-01 12:16:18 -07:00
chrislu
bee3a7c798
fix tests
2022-07-01 12:09:42 -07:00
Chris Lu
ddb0f6bc74
Merge pull request #3258 from kmlebedev/issues_3255
...
volumeServer.evacuate fix select empty node
2022-07-01 02:36:30 -07:00
Konstantin Lebedev
3d5f1cbc5e
fix pagination
2022-07-01 14:17:49 +05:00
chrislu
06a8b174b5
also remove Sync() for idx file
2022-06-30 13:50:53 -07:00
chrislu
52580743b9
stuck with file.Sync()
...
starting to get weed/storage () - (master) > go test ./...
I0630 12:19:54 65819 needle_map_metric_test.go:26] FileCount expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:27] DeletedSize expected 1648 actual 1648
I0630 12:19:54 65819 needle_map_metric_test.go:28] ContentSize expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:29] DeletedCount expected 1648 actual 1959
I0630 12:19:54 65819 needle_map_metric_test.go:30] MaxFileKey expected 10000 actual 10000
I0630 12:19:54 65819 volume_loading.go:136] loading index /var/folders/jv/0rlhdck17jzgt7b3hcnq63mc0000gp/T/TestCompaction909350722/001/1.idx to memory
I0630 12:20:36 65819 volume_vacuum.go:98] Committing volume 1 vacuuming...
panic: test timed out after 10m0s
goroutine 61 [running]:
testing.(*M).startAlarm.func1()
/usr/local/go/src/testing/testing.go:2029 +0x8e
created by time.goFunc
/usr/local/go/src/time/sleep.go:176 +0x32
goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc0000cd520, {0x1c09bde?, 0x5ab798ea5c102?}, 0x1c67bd8)
/usr/local/go/src/testing/testing.go:1487 +0x37a
testing.runTests.func1(0xc0000cd520?)
/usr/local/go/src/testing/testing.go:1839 +0x6e
testing.tRunner(0xc0000cd520, 0xc00039fcd8)
/usr/local/go/src/testing/testing.go:1439 +0x102
testing.runTests(0xc0000dadc0?, {0x25b4460, 0x6, 0x6}, {0x2855108?, 0x40?, 0x25c4b80?})
/usr/local/go/src/testing/testing.go:1837 +0x457
testing.(*M).Run(0xc0000dadc0)
/usr/local/go/src/testing/testing.go:1719 +0x5d9
main.main()
_testmain.go:57 +0x1aa
goroutine 21 [chan receive]:
github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).flushDaemon(0x0?)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:883 +0x6a
created by github.com/chrislusf/seaweedfs/weed/glog.init.0
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:410 +0x1bf
goroutine 55 [syscall]:
syscall.syscall(0x44ac0f96?, 0xc00007ad80?, 0xc00098b480?, 0x10588df?)
/usr/local/go/src/runtime/sys_darwin.go:22 +0x4e
syscall.fcntl(0x100000001?, 0xc00098b4b0?, 0x1069a7c?)
/usr/local/go/src/syscall/zsyscall_darwin_amd64.go:319 +0x30
internal/poll.(*FD).Fsync.func1(...)
/usr/local/go/src/internal/poll/fd_fsync_darwin.go:18
internal/poll.ignoringEINTR(...)
/usr/local/go/src/internal/poll/fd_posix.go:74
internal/poll.(*FD).Fsync(0xc00038f1e0?)
/usr/local/go/src/internal/poll/fd_fsync_darwin.go:17 +0xfc
os.(*File).Sync(0xc00012a030)
/usr/local/go/src/os/file_posix.go:168 +0x4e
github.com/chrislusf/seaweedfs/weed/storage/backend.(*DiskFile).Sync(...)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/backend/disk_file.go:84
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).makeupDiff(0xc0000bb440, {0xc000633a40, 0x52}, {0xc000633aa0, 0x52}, {0xc000633b00, 0x52}, {0xc000633b60, 0x52})
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:295 +0x12fa
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).CommitCompact(0xc0000bb440)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:119 +0x3e8
github.com/chrislusf/seaweedfs/weed/storage.TestCompaction(0xc00025a000)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum_test.go:89 +0x305
testing.tRunner(0xc00025a000, 0x1c67bd8)
/usr/local/go/src/testing/testing.go:1439 +0x102
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1486 +0x35f
goroutine 56 [chan receive, 10 minutes]:
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker.func1()
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:244 +0x41a
created by github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:234 +0x56
FAIL github.com/chrislusf/seaweedfs/weed/storage 600.194s
? github.com/chrislusf/seaweedfs/weed/storage/backend [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/backend/memory_map (cached)
? github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/erasure_coding (cached)
? github.com/chrislusf/seaweedfs/weed/storage/idx [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/needle (cached)
ok github.com/chrislusf/seaweedfs/weed/storage/needle_map (cached)
ok github.com/chrislusf/seaweedfs/weed/storage/super_block (cached)
? github.com/chrislusf/seaweedfs/weed/storage/types [no test files]
? github.com/chrislusf/seaweedfs/weed/storage/volume_info [no test files]
FAIL
2022-06-30 12:32:55 -07:00
Konstantin Lebedev
5ed8165161
fix logic
...
add option targetServer
https://github.com/chrislusf/seaweedfs/issues/3255
2022-06-30 17:58:24 +05:00
石昌林
200a862701
fixed volume xx not found caused by missing VolumeLocation events
...
When the requested master node is the leader, `VolumeLocation` or `ClusterNodeUpdate` may be returned here. If it is `VolumeLocation`, the update will be performed while resetting the vidMap, otherwise the event will be lost
2022-06-30 13:41:56 +08:00
Chris Lu
4ae9884a6e
Merge pull request #3254 from shichanglin5/fix_subscribe_meta
...
[Critical] Avoid synchronizing from scratch due to err
2022-06-29 12:36:28 -07:00
Chris Lu
b98556cff5
Merge pull request #3253 from kmlebedev/constMultipartUploadsFolder
...
use const multipart uploads folder
2022-06-29 12:09:03 -07:00
石昌林
2b5b37231b
Make sure that onChangeEvent is called successfully before updating lastTsNs
2022-06-29 23:57:07 +08:00
石昌林
e17e6ed542
Avoid synchronizing from scratch due to err
2022-06-29 23:31:04 +08:00
Konstantin Lebedev
bcbdc4cb37
use const multipart uploads folder
...
avoid error bucket NotEmpty if multipart uploads folder exist
2022-06-29 16:21:16 +05:00
chrislu
a8abab2412
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-29 02:44:14 -07:00
chrislu
449ecc8890
remove max connection age
...
related to https://github.com/chrislusf/seaweedfs/pull/3226
2022-06-29 02:44:12 -07:00
Chris Lu
c6e6e303db
Merge pull request #3200 from lapshin-vitaly/bufix/validate-tags-on-copy
...
validate tags on copy object and add regex for validating tags
2022-06-29 01:37:14 -07:00
Lapshin Vitaliy
7c3d9d0535
remove println
2022-06-29 10:43:02 +03:00
Lapshin Vitaliy
61b2e3f9a2
fix return
2022-06-28 18:31:46 +03:00
Lapshin Vitaliy
606252472c
fix return
2022-06-28 18:24:03 +03:00
Lapshin Vitaliy
e969370913
fix parseTagging function and add test for validate tags
2022-06-28 14:48:55 +03:00
Chris Lu
a79bf9d92a
Merge pull request #3252 from guol-fnst/update_tbk
...
remove hdfs, add tikv for full version update
2022-06-27 22:37:41 -07:00
guol-fnst
74c9f39d48
remove hdfs, add tikv for full version update
2022-06-28 12:35:07 +08:00
石昌林
e1b94eb6b9
fix error: concurrent map writes when add or remove cluster node
2022-06-28 10:34:59 +08:00
chrislu
4b1f48a399
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-27 12:40:50 -07:00
chrislu
4a65159250
fix reading time
2022-06-27 12:40:47 -07:00
xdadrm
3a42269da8
Fixes WebDAV 0-bytes files
...
Fixes the issue where files created via WebDAV show as 0-bytes size when read via fuse.
2022-06-27 20:19:21 +02:00
Chris Lu
4be406a095
Merge pull request #3249 from kmlebedev/issues_2589
...
avoid set currentMaster svc.local k8s discovery service domains
2022-06-27 11:05:07 -07:00
Konstantin Lebedev
9bff097b01
currentMaster redirected
2022-06-27 21:55:16 +05:00
Konstantin Lebedev
6c20a3b622
avoid set currentMaster k8s svc.local discoveruy service domains
...
https://github.com/chrislusf/seaweedfs/issues/2589
2022-06-27 21:47:05 +05:00
Lapshin Vitaliy
1a5981d583
fix test
2022-06-27 18:46:21 +03:00
garenchan
8aa19577f4
fix 3238: handle errors for GenerateDirUuid method
2022-06-27 22:04:50 +08:00
Lapshin Vitaliy
d7c3493d15
Merge branch 'master' into bufix/validate-tags-on-copy
2022-06-27 13:53:57 +03:00
chrislu
b9f7b6fb9a
adjust log message
2022-06-26 23:12:16 -07:00
Chris Lu
fdd880b111
Merge pull request #3237 from shichanglin5/remove_query_collection
...
Remove the collection query param of s3api and let the collection be …
2022-06-26 23:02:11 -07:00
石昌林
15bfc60a88
Remove the collection query param of s3api and let the collection be determined by the filer
2022-06-27 13:47:53 +08:00
yulai.li
af23e63e3f
Improve filer command help, add supported filer store list
2022-06-27 12:09:16 +08:00
chrislu
40a9634c3d
3.13
2022-06-26 20:13:52 -07:00
Chris Lu
cc01f71752
Merge pull request #3233 from blacktear23/filerstore-tikv
...
Add TiKV Filerstore back
2022-06-26 20:12:18 -07:00
yulai.li
824cbe32be
Make tikv filer enable/disable by build tags
2022-06-27 10:57:24 +08:00
chrislu
9dfea44aeb
fix unit
2022-06-26 12:21:38 -07:00
chrislu
3142e77eca
mount: stats report physical size instead of logical size
2022-06-26 12:14:34 -07:00
yulai.li
13159ad0d8
Merge branch 'master' into filerstore-tikv
2022-06-26 22:44:01 +08:00
yulai.li
46e0b629e5
Update tikv client version and add one PC support
2022-06-26 22:43:37 +08:00
chrislu
f77eda4b4f
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-25 09:22:52 -07:00
chrislu
3ce46c3566
fix entry full url lookup
2022-06-25 09:22:49 -07:00
Chris Lu
dc59ccd110
Merge pull request #3228 from shichanglin5/fix_volumeNotFound
...
When the connection with the leader is disconnected, the vidMap shoul…
2022-06-24 23:06:13 -07:00
石昌林
f1ea906c13
Code comment optimization
2022-06-25 11:56:09 +08:00
石昌林
0c8e033e68
When the connection with the leader is disconnected, the vidMap should not be reset immediately, but should be reset after connecting to a new leader
2022-06-24 17:41:46 +08:00
chrislu
9c517d2b35
masterclient: fallback to directly querying master in case of missing volume id location
2022-06-24 02:08:57 -07:00
Chris Lu
280e33092c
Merge pull request #3226 from kmlebedev/advancedtls
...
Advancedtls
2022-06-24 01:34:58 -07:00
chrislu
509a9047db
test compact map with snowflake sequencer
2022-06-23 21:45:51 -07:00
chrislu
217d7dc377
verify seq is always unique
2022-06-23 21:22:05 -07:00
Konstantin Lebedev
2e782dfdcb
rm defer comments
2022-06-24 01:50:13 +05:00
Konstantin Lebedev
7efaafb0ee
fix Authenticate
2022-06-24 01:42:04 +05:00
Konstantin Lebedev
b0aa51d7ef
enable require client cert
2022-06-24 00:29:23 +05:00
Konstantin Lebedev
ea7cdb8b0e
seperate option
2022-06-24 00:26:56 +05:00
Konstantin Lebedev
5b388ed6c1
initial advancedtls
2022-06-23 23:32:15 +05:00
chrislu
0c13fc873c
fix compilation
2022-06-23 00:45:11 -07:00
chrislu
96496d5286
master: broadcast new volume locations to clients to avoid possible racing condition
...
fix https://github.com/chrislusf/seaweedfs/issues/3220
2022-06-23 00:41:33 -07:00
chrislu
52c44d646e
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-22 00:34:27 -07:00
chrislu
530da0cc89
remove unused code
2022-06-22 00:34:25 -07:00
famosss
8a6e7a7df8
Merge branch 'chrislusf:master' into master
2022-06-22 09:58:56 +08:00
zzq09494
9df5ad5309
fix: vacuum create a lot of connections quickly
2022-06-22 09:57:22 +08:00
Chris Lu
d652ce1a36
Merge pull request #3216 from kmlebedev/mk_raft_basedir
...
avoid no such raft date directory
2022-06-21 12:21:30 -07:00
chrislu
6b3beb709d
fix format
2022-06-21 12:21:02 -07:00
Konstantin Lebedev
c1c8dad677
avoid no such raft date directory
...
https://github.com/chrislusf/seaweedfs/issues/3214
2022-06-21 13:47:51 +05:00
chrislu
4dc27e1ed5
3.12
2022-06-20 19:07:00 -07:00
chrislu
492da3dbce
master: put metadata under instance specific folder
2022-06-20 19:04:49 -07:00
chrislu
41dfe27102
Revert "remove max connection age"
...
This reverts commit b9b684194f
2022-06-20 14:49:39 -07:00
chrislu
2295766e2c
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-20 14:16:18 -07:00
chrislu
1d0c53ea56
remote storage: stop supporting hdfs as a remote storage
2022-06-20 14:15:59 -07:00
Stephan
1eb7826909
Fix link to osxfuse github page
2022-06-20 22:36:07 +02:00
chrislu
bf5a5abfb1
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-20 13:18:11 -07:00
chrislu
69a2e503ab
non-recursive directory deletion
...
fix a problem during git checkout a different version, where updated directories are deleted even though the directory is not empty.
2022-06-20 13:18:08 -07:00
chrislu
76f26c4b99
mount: delete target entry first during move
2022-06-20 13:15:02 -07:00
Chris Lu
460c7cf68e
Merge pull request #3203 from shichanglin5/volume_copy_preallocate
2022-06-20 07:41:45 -07:00
石昌林
81f7f08708
Determine whether to preallocate according to the master configuration before executing copy volume
2022-06-20 21:12:44 +08:00
石昌林
be5c901bea
Fix global counter key is incorrect & Add read lock when reading counter map to avoid memory problems
2022-06-20 19:16:53 +08:00
Lapshin Vitaliy
5f5fd0bc48
validate tags on copy object and add regex for validating tags
2022-06-20 11:32:58 +03:00
Chris Lu
9e2d6e897e
Merge pull request #3198 from guol-fnst/fix_statuscode
...
AbortMultipartUploadHandler should return 204 instead of 200
2022-06-19 22:57:13 -07:00
LHHDZ
8db9f13bc6
Merge branch 'master' into circuit_breaker
2022-06-20 13:42:18 +08:00
guol-fnst
1d77deccd0
AbortMultipartUploadHandler should return 204 instead of 200
...
https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html
2022-06-20 13:26:22 +08:00
石昌林
9e036df356
remove go.uber.org/atomic
2022-06-20 12:35:29 +08:00
geekboood
fdacd94af5
fix: invalid chunk data when failed to read manifests
2022-06-19 16:54:04 +08:00
石昌林
3dd60529c5
some code optimizations
2022-06-17 19:07:39 +08:00
石昌林
37df209195
add some unit tests and some code optimizes
2022-06-17 17:11:18 +08:00
Chris Lu
d28f8bb88f
Merge pull request #3192 from guol-fnst/weed_update
...
remove "weed" from UsageLine, or weed will not show usage of weed update
2022-06-16 22:58:29 -07:00
guol-fnst
c7c20881f5
remove "weed" from UsageLine, or weed will not show usage of weed update
2022-06-17 12:47:04 +08:00
chrislu
625fd16a2e
reduce upfront memory usage for low density volume
2022-06-16 15:39:29 -07:00
Chris Lu
5f8d6b5f13
Merge pull request #3184 from guol-fnst/weed_update
...
Weed update
2022-06-16 10:02:45 -07:00
guol-fnst
cd078471c4
fix path bug on windows
...
fix usage
2022-06-16 17:24:56 +08:00
chrislu
aaa9938353
Revert "adjust conditions"
...
This reverts commit e024586ff1
2022-06-16 01:26:36 -07:00
guol-fnst
995e221e97
remove -output, add -dir,-name for more flexibility
2022-06-16 16:07:32 +08:00
chrislu
e024586ff1
adjust conditions
2022-06-15 23:19:39 -07:00
Chris Lu
82f3bcc65e
Merge pull request #3081 from paochiang/volume_upload_limit_fix
...
exclude replication from the concurrentUploadLimitMB
2022-06-15 23:16:44 -07:00
liubaojiang
df0ce31a2e
add condition when inFlightUploadDataLimitCond signal
2022-06-16 14:07:11 +08:00
zzq09494
62f74f5e3c
add bucket label to s3 prometheus metrics
2022-06-16 13:21:25 +08:00
guol-fnst
4bc8792643
support specific version of weed
2022-06-16 11:40:23 +08:00
guol-fnst
1dbb925d29
remove unused function
...
remove log when weed is up to date
add check whether weed is full version
2022-06-16 11:40:20 +08:00
liubaojiang
3076ac101e
move vs.concurrentUploadLimit != 0 out of the lock
2022-06-16 09:58:44 +08:00
Chris Lu
097a12dfb4
Merge pull request #3187 from ningfdx/remote
2022-06-15 06:34:32 -07:00
Chris Lu
55b9f281d1
Merge pull request #3185 from kmlebedev/logging_err_process_range_req
2022-06-15 06:33:33 -07:00
石昌林
78b3728169
add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations
...
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply
# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply
# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply
# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply
# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
2022-06-15 21:07:55 +08:00
ningfd
338705f375
fix(wdclient): GetLocations return
2022-06-15 19:20:13 +08:00
Konstantin Lebedev
31a929f7f2
logging processRangeRequest errors
2022-06-15 15:36:43 +05:00
Chris Lu
4eeeb5f50d
Merge pull request #3180 from blacktear23/issue-3149
2022-06-15 00:55:20 -07:00
yulai.li
358c0c0b37
Fix: provide filer.ui.deleteDir options for server command
2022-06-15 15:53:31 +08:00
Guo Lei
3df84cabf6
Merge branch 'chrislusf:master' into weed_update
2022-06-15 15:28:40 +08:00
guol-fnst
695e8c7555
fix minor bug
2022-06-15 15:26:04 +08:00
zhihao.qu
4d0d1848c6
fix(filer.sync): modify clientName format : from -> to
2022-06-15 13:33:20 +08:00
yulai.li
b90d213a87
Change ui.deleteDir default to true
2022-06-15 11:37:37 +08:00
zhihao.qu
42d04c581b
feat(filer.sync): add metricsServer in filer.sync.
...
Metrics include:
(1) the offset of the filer.sync
(2) the last send timestamp of the filer subscription
2022-06-15 11:33:18 +08:00
chrislu
d4ef06cdcf
3.11
2022-06-14 12:14:52 -07:00
chrislu
b347b2fb54
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-14 11:16:51 -07:00
chrislu
36d2756978
volume: avoid writing too much data for range requests in large files
...
related to https://github.com/chrislusf/seaweedfs/issues/3178
2022-06-14 11:16:47 -07:00
yulai.li
28cda854eb
Add filer command line parameter to let Filer UI show/hide directory delete button
2022-06-15 00:30:49 +08:00
zhihao.qu
14d82c3dea
feat(filer.sync): add offset to path.
2022-06-14 19:46:02 +08:00
guol-fnst
78fc9cdfa0
add "copied from https://github.com/restic/restic/tree/master/internal/selfupdate "
2022-06-14 19:10:54 +08:00
guol-fnst
322dbcf372
add update feature
2022-06-14 19:01:57 +08:00
Chris Lu
b22ca85fbb
Merge pull request #3169 from shichanglin5/fix_auth
...
fix: When there is no access permission configured before startup, th…
2022-06-12 23:40:42 -07:00
shichanglin5
f01dd27752
fix: When there is no access permission configured before startup, the authentication does not take effect after configuring the permission after startup
2022-06-13 13:23:26 +08:00
chrislu
e437ec5a68
3.10
2022-06-12 19:30:27 -07:00
chrislu
d12f431d98
collect volume server status
2022-06-12 11:56:23 -07:00
chrislu
444ac21050
go fmt
2022-06-11 09:51:11 -07:00
chrislu
a74e926063
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-11 09:51:02 -07:00
chrislu
61b8c9c361
remote object store gateway: disable tagging for backblaze
2022-06-11 09:50:59 -07:00
Chris Lu
0a8b3948db
Merge pull request #3165 from blacktear23/reload-bug-fix
2022-06-10 07:47:43 -07:00
yulai.li
83f8cdd5cb
Bug Fix: force reload filer page when upload/delete/rename/create directory
2022-06-10 22:39:21 +08:00
guol-fnst
eabf4d4e64
fix typo
2022-06-10 22:03:11 +08:00
Chris Lu
36757d28a8
Merge pull request #3163 from kmlebedev/metric_replica_placement_mismatch
...
stats master_replica_placement_mismatch
2022-06-10 06:44:16 -07:00
chrislu
9f8b72a54d
Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID"
...
This reverts commit 37da68931900d53c34c4
2022-06-10 06:38:17 -07:00
Konstantin Lebedev
fb57aa431c
stats master_replica_placement_mismatch
2022-06-10 15:30:40 +05:00
shichanglin5
a63559ee6c
fix: fix command prompt information
2022-06-10 12:46:10 +08:00
Chris Lu
37da689319
Merge pull request #3159 from shichanglin5/_duplicateUUID
...
perf: Optimized volume handling duplicateUUID logic to avoid quitting…
2022-06-09 18:01:02 -07:00
shichanglin5
f5b0c04b14
perf: Optimized volume handling duplicateUUID logic to avoid quitting when volume is actualy normal
...
Under normal circumstances, there will be no problems, but when the
master is debugged in the local environment, the volume client cannot
communicate with the master normally, so the sendHeartBeat logic is
restarted, and a new connection is created to report the heartbeat. If
the master has not cleared the uuid of the volume at this time, then The
master will respond to volume duplicateUUIDS, and the volume service
will exit, but in fact the uuid of the volume is not duplicated
2022-06-09 20:41:16 +08:00
zhihao.qu
cd5cca36a4
feat(filer.sync): add fromTsMs. Extract signature from doSubscribeFilerMetaChanges
2022-06-09 10:53:19 +08:00
Chris Lu
4a5135961f
Merge pull request #3154 from kmlebedev/s3_test_invalid_range
2022-06-08 01:45:40 -07:00
Konstantin Lebedev
49f058da85
fix s3 tests:
...
ranged_request_invalid_range
ranged_request_empty_object
2022-06-08 13:25:13 +05:00
chrislu
8a1166c021
auto remove previous leftover socket
2022-06-08 00:32:09 -07:00
chrislu
633a4aed18
filer: fix customizable local socket file name
...
fix https://github.com/chrislusf/seaweedfs/issues/3147
2022-06-08 00:15:28 -07:00
chrislu
acbc0286e2
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-07 22:33:38 -07:00
chrislu
ebd6097e6f
fix: sequence read rate is slow when use filer.ChunkReadAt.ReadAt #3074
...
related to https://github.com/chrislusf/seaweedfs/pull/3074
2022-06-07 22:33:35 -07:00
Chris Lu
f43ec9f363
Merge pull request #3130 from kmlebedev/fix_rm_parent_dir_via_nextcloud
2022-06-07 08:56:55 -07:00
Konstantin Lebedev
c07820178f
fix s3 tests
...
bucket_list_delimiter_prefix
bucket_list_delimiter_prefix_underscore
bucket_list_delimiter_prefix_ends_with_delimiter
2022-06-07 14:43:10 +05:00
chrislu
78be3b0f6c
fix compilation
2022-06-06 10:09:21 -07:00
chrislu
4fd5f96598
filer: remove replication, collection, disk_type info from entry metadata
...
these metadata can change and are not used
2022-06-06 00:39:35 -07:00
chrislu
cbf46de5f4
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-06 00:15:40 -07:00
chrislu
8402af0882
filer: detect ttl based on fs configure setting
2022-06-06 00:15:37 -07:00
ningfd
f32142f6f5
add disableXAttr in mount option
2022-06-06 14:09:01 +08:00
chrislu
4a046e4de7
3.09
2022-06-05 19:00:24 -07:00
chrislu
11f1e7996b
fix test
2022-06-05 18:50:41 -07:00
chrislu
d65bb2c6df
mount: file handle locks entry better
...
related to https://github.com/chrislusf/seaweedfs/issues/2952
2022-06-05 18:15:06 -07:00
chrislu
746092a60b
add a comment
2022-06-05 16:44:07 -07:00
chrislu
056c480eb0
volume: checksum remove the hashing step in Value()
2022-06-05 15:24:02 -07:00
chrislu
ecef844dfc
stream read large files
2022-06-05 11:54:04 -07:00
chrislu
48cca4e54f
remove unused logic
2022-06-04 19:11:06 -07:00
chrislu
685643d506
needle read into a writer
2022-06-04 19:10:52 -07:00
chrislu
663bc5dc23
add needle reading in chunks
2022-06-04 18:15:39 -07:00
chrislu
18b29f709c
refactoring, split file
2022-06-03 23:31:14 -07:00
chrislu
39c3cb04af
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-03 08:45:32 -07:00
chrislu
f97acdd489
volume.fix.replication fix retry logic
...
fix https://github.com/chrislusf/seaweedfs/issues/3136
2022-06-03 08:45:29 -07:00
creeew
02ae102731
fix filer.sync missing source srv uploaded files to target when target down
2022-06-02 01:28:47 +08:00
Konstantin Lebedev
b86628f85d
fix show empty dir
...
https://github.com/chrislusf/seaweedfs/issues/3086
2022-06-01 20:10:52 +05:00
chrislu
6793bc853c
help message when in simulation mode
2022-05-31 14:48:46 -07:00
chrislu
89948a373b
fix error reporting for "Need to a hdd disk type"
...
related to https://github.com/chrislusf/seaweedfs/issues/3128
2022-05-31 12:43:55 -07:00
chrislu
8a49240d64
3.08
2022-05-31 11:57:41 -07:00
chrislu
376b005ad3
fix wrong logic
2022-05-31 11:55:58 -07:00
chrislu
fbd99d53c1
3.07
2022-05-31 00:36:24 -07:00
chrislu
27732ecfa4
move s3 related constants from package http to s3_constants
2022-05-30 22:57:41 -07:00
chrislu
f4a6da6cb2
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-30 22:47:29 -07:00
chrislu
596c3860ca
use final destination to resolve fs configuration
...
related to https://github.com/chrislusf/seaweedfs/issues/3075
2022-05-30 22:47:26 -07:00
Chris Lu
ca01ce0524
Merge pull request #3122 from chrislusf/filer-sync-with-peers
...
Filer bootstrap from peers
2022-05-30 21:38:31 -07:00
chrislu
6adc42147f
fresh filer store bootstrap from the oldest peer
2022-05-30 21:27:48 -07:00
chrislu
490e0850bf
filer.meta.tail add one example of using untilTimeAgo
2022-05-30 20:24:59 -07:00
liubaojiang
f0ee3e6f21
reduce the scope of inFlightUploadDataLimitCond lock
2022-05-31 09:49:07 +08:00
chrislu
c59068d0f3
refactor
2022-05-30 16:28:36 -07:00
chrislu
682382648e
collect cluster node start time
2022-05-30 16:23:52 -07:00
chrislu
968ca95b49
filer.meta.tail: support untilTimeAgo for a range
2022-05-30 16:16:23 -07:00
chrislu
f214dfb1f5
stop when in memory log is done
2022-05-30 15:25:21 -07:00
chrislu
aece35a64f
stop when on disk log is done
2022-05-30 15:20:51 -07:00
chrislu
a2b101a737
subscribe metadata between a range
2022-05-30 15:04:19 -07:00
chrislu
8902fa6ff6
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-30 13:28:32 -07:00
Chris Lu
730c9cf601
Merge pull request #3114 from kmlebedev/nextcloud
...
Create folders in s3 via nextcloud
2022-05-30 13:10:10 -07:00
chrislu
5b9347c938
typo
2022-05-30 12:37:41 -07:00
Konstantin Lebedev
af562e1a20
rm log info
2022-05-30 23:26:26 +05:00
Konstantin Lebedev
62e5e3822c
list self dir
...
https://github.com/chrislusf/seaweedfs/issues/3086
2022-05-30 18:09:08 +05:00
Konstantin Lebedev
26f3646961
show fake dirs
...
https://github.com/chrislusf/seaweedfs/issues/3086
2022-05-30 15:03:27 +05:00
chrislu
b8f3db0d46
s3: keep auth enabled in case identities are set to empty
...
fix https://github.com/chrislusf/seaweedfs/issues/3084
2022-05-29 19:00:03 -07:00
chrislu
81d6159290
volume: report error if a volume has nil data backend
...
fix https://github.com/chrislusf/seaweedfs/issues/3105
2022-05-29 16:59:30 -07:00
chrislu
ca16fbf0ef
shell: cluster.ps display filer group
2022-05-29 16:37:14 -07:00
chrislu
9219664505
reset sync offset if peer filer resets
...
fix https://github.com/chrislusf/seaweedfs/issues/3089
2022-05-29 13:11:24 -07:00
Konstantin Lebedev
54356211c2
list empty folder
2022-05-27 13:17:18 +05:00
Kaiwalya Joshi
4598581741
fix: Minor spelling fixes on FilerStorePathTranslator.
2022-05-25 01:11:02 -07:00
chrislu
5b8b022985
remove unused parameter
2022-05-24 18:52:04 -07:00
Robert Coelho
0e6e72d462
mount: ReadDir return EIO on EnsureVisited err
2022-05-24 13:02:59 -04:00
Robert Coelho
1fabbe8a25
mount: cleanup ReadDir rewrite's branches to not assume offsets
2022-05-24 12:48:32 -04:00
Robert Coelho
cb422d96f7
mount: rewrite ReadDir to respect input.Offset to fix partial results
2022-05-24 12:16:58 -04:00
Robert Coelho
05602167f4
mount: do not mark meta_cache when there is an err
2022-05-24 12:13:15 -04:00
chrislu
7e25a2d416
reduce busy waiting when reading metadata logs
2022-05-24 00:23:53 -07:00
chrislu
2f846777bb
3.06
2022-05-23 01:21:48 -07:00
chrislu
a5cdcc7e63
fix reading files larger than 20GB due to manifest resolving bug
2022-05-23 01:16:10 -07:00
chrislu
59cc3c0bfd
fix reading files larger than 20GB due to manifest resolving bug
...
this is introduced since Mar 08 2022
2022-05-23 01:14:56 -07:00
chrislu
acc318e12b
3.05
2022-05-22 21:43:59 -07:00
liubaojiang
076e48a676
add inflight upload data wait timeout
2022-05-21 10:38:08 +08:00
liubaojiang
71b2e6223e
add inFlightUploadDataLimitCond signal
2022-05-20 15:27:06 +08:00
liubaojiang
1a41691b4c
exclude replication from the concurrentUploadLimitMB
2022-05-20 14:40:05 +08:00
Chris Lu
cb64983c6b
Merge pull request #3079 from ztinpn/hotfix_fsync
2022-05-19 10:33:42 -07:00
tianzhang
66747ee9c9
hotfix_fsync fix fsync
2022-05-19 23:17:17 +08:00
Chris Lu
8efe55f981
Merge pull request #3059 from guol-fnst/avoid_dup_vol
...
avoid loading duplicated volume directory
2022-05-17 23:48:57 -07:00
guol-fnst
8e2426a879
revert changes of pb file
...
orignal protoc-gen-go-grpc verion is 1.1.0
2022-05-18 11:52:14 +08:00
guol-fnst
b12944f9c6
fix naming convention
...
notify volume server of duplicate directoris
improve searching efficiency
2022-05-17 15:41:49 +08:00
guol-fnst
076595fbdd
just exit in case of duplicated volume directories were loaded
2022-05-17 15:41:49 +08:00
guol-fnst
8fab39e775
rename UUID file
...
fix typo
move locationUUID into DiskLocation
2022-05-17 15:41:47 +08:00
Chris Lu
72e7dcde51
Merge pull request #3060 from natmaka/master
2022-05-16 07:42:58 -07:00
Nat Makarevitch
b72f0634d7
minor (typos...), done while reading around
2022-05-16 22:11:33 +08:00
guol-fnst
de6aa9cce8
avoid duplicated volume directory
2022-05-16 19:33:51 +08:00
Konstantin Lebedev
44f53ceda6
fix collectionIsMismatch charset
2022-05-16 13:23:23 +05:00
Konstantin Lebedev
10d435f2c2
fix skip loop
2022-05-16 13:16:27 +05:00
Konstantin Lebedev
279053572c
avoid delete volume replica if collection mismatch
2022-05-16 13:07:05 +05:00
chrislu
9ff0d99002
3.04
2022-05-15 21:32:21 -07:00
chrislu
866fed1eab
avoid grpc name conflict
...
fix https://github.com/chrislusf/seaweedfs/issues/3055
2022-05-15 21:31:29 -07:00
chrislu
cd2d10118c
3.03
2022-05-15 18:17:07 -07:00
chrislu
271b5aed96
shell: volume.fsck add a note for -reallyDeleteFromVolume option
2022-05-15 11:07:04 -07:00
chrislu
4076d74813
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-15 00:43:39 -07:00
chrislu
29198720f2
s3: add grpc server to accept configuration changes
2022-05-15 00:43:37 -07:00
Chris Lu
ea4918394c
Merge pull request #3047 from shichanglin5/master
...
fix the problem of metadata and tagging loss when files are copied
2022-05-13 22:52:48 -07:00
shichanglin5
688d55488c
test(s3api_object_copy_handlers_test.go): some unit tests have been added to the processMetadata & processMetadataBytes methods of s3api_object_copy_handlers.go
2022-05-14 10:40:29 +08:00
chrislu
735038b2c1
backup do not need to use preallocation
...
fix https://github.com/chrislusf/seaweedfs/issues/3044
2022-05-13 13:46:52 -07:00
shichanglin5
1166dead00
fix the problem of metadata and tagging loss when files are copied
...
by adding processing of metadata and tagging in s3 api CopyObject (judging whether to copy or overwrite according to the directive header)
2022-05-13 19:46:20 +08:00
Konstantin Lebedev
d12a423aa0
refactor
...
add some tests
https://github.com/chrislusf/seaweedfs/pull/2996
2022-05-12 12:23:02 +05:00
Konstantin Lebedev
f523aed3c9
fixes from comments
...
https://github.com/chrislusf/seaweedfs/pull/2996
2022-05-11 21:50:09 +05:00
chrislu
bff1ccc1de
fix compilation
2022-05-11 00:52:15 -07:00
chrislu
76da688176
adjust log level
2022-05-11 00:45:19 -07:00
chrislu
3e20336bde
fix atomic add int64
...
fix https://github.com/chrislusf/seaweedfs/issues/3038#issuecomment-1123269831
2022-05-11 00:44:35 -07:00
chrislu
a85720f0f6
remove deprecated function
2022-05-10 00:02:23 -07:00
chrislu
25559eeb92
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-10 00:01:46 -07:00
chrislu
791c29ae93
remove deprecated function
...
WithQueryCachePolicyKeepInCache is deprecated and already default behavior
2022-05-10 00:01:43 -07:00
qiang.zeng
06cb337b82
fix typo
2022-05-09 14:21:16 +08:00
chrislu
a4ca3ed1f0
3.02
2022-05-08 22:28:32 -07:00
chrislu
8bc16f1149
rename
2022-05-06 23:38:22 -07:00
chrislu
139e039c44
filer.sync: pass attributes for mount
...
fix https://github.com/chrislusf/seaweedfs/issues/3012
2022-05-06 03:54:12 -07:00
chrislu
10207a48a4
how to setup local test env
2022-05-06 02:25:56 -07:00
chrislu
db33b03735
add comments
2022-05-06 02:19:47 -07:00
chrislu
e350ba8896
larger test range
2022-05-06 02:17:28 -07:00
chrislu
71e0bcc5d5
a bit optimization
2022-05-06 02:11:58 -07:00
chrislu
b2a43dd044
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-06 01:51:28 -07:00
chrislu
d5fe5d12b7
filer: etcd fix pagination
...
fix https://github.com/chrislusf/seaweedfs/issues/3016
2022-05-06 01:51:19 -07:00
Chris Lu
a69bf53405
Merge pull request #2996 from kmlebedev/ydb
...
ydb filer store
2022-05-05 15:34:37 -07:00
Konstantin Lebedev
e6d2cb5911
ydb add empty doc file
2022-05-05 11:12:14 +05:00
chrislu
ef6c6c450e
avoid fatal error if port is already in use
2022-05-04 14:55:14 -07:00
chrislu
f7366a9668
skip unix socket mode for windows
...
fix https://github.com/chrislusf/seaweedfs/issues/3013
2022-05-04 10:14:34 -07:00
Konstantin Lebedev
f58adaab25
build tag ydb since the growth of the binary is 5 mb
2022-05-03 23:20:09 +05:00
Konstantin Lebedev
f127b326bf
add options to scaffold
2022-05-03 22:54:31 +05:00
Konstantin Lebedev
7b544576af
refactor
2022-05-03 20:16:00 +05:00
chrislu
f17cd0d5cd
return false if not found
...
fix https://github.com/chrislusf/seaweedfs/issues/3011
2022-05-03 07:18:34 -07:00
chrislu
b201edb9df
fix wrong assignment
2022-05-03 07:12:33 -07:00
Konstantin Lebedev
04d5dff6bb
add support native ttl
2022-05-03 17:52:23 +05:00
Konstantin Lebedev
cb3c7a3cdb
enable query cache policy instead of prepare
2022-05-03 16:03:10 +05:00
Konstantin Lebedev
8342f651f3
fix scanNamed
2022-05-03 15:18:28 +05:00
chrislu
9271866d1e
fix segmentation violation
...
fix https://github.com/chrislusf/seaweedfs/issues/3000
2022-05-02 23:56:11 -07:00
Konstantin Lebedev
2d27afb515
Merge branch 'new_master' into ydb
2022-05-03 10:47:39 +05:00
chrislu
76148ce0f7
use storage_class for backend configuration
...
fix https://github.com/chrislusf/seaweedfs/issues/3008
2022-05-02 12:20:01 -07:00
Konstantin Lebedev
d8925b4e83
Merge branch 'new_master' into ydb
...
# Conflicts:
# go.mod
# go.sum
2022-05-03 00:13:57 +05:00
Konstantin Lebedev
7640e650e5
fix queries
2022-05-03 00:11:37 +05:00
Konstantin Lebedev
319d300d48
fix createDB
2022-05-02 22:23:07 +05:00
Konstantin Lebedev
0dc44dda63
ydb do Tx or DB
2022-05-02 15:33:29 +05:00
Konstantin Lebedev
a3e48831f3
ydb BucketAware interface
2022-05-02 12:42:20 +05:00
chrislu
73961e24d8
3.01
2022-05-01 23:33:45 -07:00
chrislu
00c1dfec4f
go fmt
2022-05-01 23:16:29 -07:00
chrislu
ce1e8e58ae
fix test
2022-05-01 23:15:58 -07:00
chrislu
2c0fdf3693
fix tests
2022-05-01 23:14:04 -07:00
chrislu
94635e9b5c
filer: add filer group
2022-05-01 21:59:16 -07:00
Konstantin Lebedev
50c4f62ed4
ydb-go-sdk move to v3
2022-05-02 02:07:47 +05:00
Konstantin Lebedev
21033ff4c3
refactor use const CountEntryChunksForGzip
2022-05-01 22:28:55 +05:00
Konstantin Lebedev
ec0ed41e37
ydb kv interface
2022-05-01 21:20:37 +05:00
chrislu
4bd6bea429
fix test
2022-05-01 09:02:01 -07:00
Konstantin Lebedev
6a052f6ff2
ydb Sql interface
2022-05-01 20:33:03 +05:00
Konstantin Lebedev
275c3bb19c
ydb initial
...
https://github.com/chrislusf/seaweedfs/issues/2942
2022-05-01 17:28:58 +05:00
chrislu
5d081d5c3c
skip if already copied to remote
...
fix https://github.com/chrislusf/seaweedfs/issues/2993
2022-04-30 20:03:04 -07:00
chrislu
1aae7a3f1b
volume.tier.upload progress starts negative #2992
...
fix https://github.com/chrislusf/seaweedfs/issues/2992
2022-04-30 18:10:01 -07:00
chrislu
ea8a7a5584
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-04-30 17:36:45 -07:00
chrislu
192983b464
s3 backend support customizing storage class
2022-04-30 17:36:40 -07:00
Konstantin Lebedev
03aec9ce7e
avoid panic if hashicorp raft not initialized
2022-04-30 13:56:35 +05:00
guosj
1a5de36257
fix return value
2022-04-29 11:19:36 +08:00
Chris Lu
4e97b6146a
Merge pull request #2982 from kmlebedev/issues/2981
...
avoid empty listMultipartUploads response
2022-04-27 17:21:44 -07:00
chrislu
b2a6111090
skip ipv6 all interfaces and localhost
...
fix https://github.com/chrislusf/seaweedfs/issues/2983
2022-04-27 17:18:09 -07:00
Konstantin Lebedev
306cf70c4a
avoid empty listMultipartUploads response
2022-04-27 19:27:44 +05:00
chrislu
70e5a1b632
volume close should wait for committing compaction
2022-04-26 23:34:05 -07:00
Chris Lu
ffe028f8d0
Merge pull request #2974 from kmlebedev/wait_volume_closed_compression
...
waite volume being closed during compression idx
2022-04-26 23:29:22 -07:00
chrislu
37ab8909b0
use two flags: v.isCompacting and v.isCommitCompacting
2022-04-26 23:28:34 -07:00
Chris Lu
b557faf175
Merge pull request #2967 from kmlebedev/vl_chk_disk_vlid
...
volume.check.disk add param volumeId
2022-04-26 22:09:06 -07:00
Chris Lu
eb49bd5638
Merge pull request #2970 from kmlebedev/vlm_fsck_forcePurging
...
delete missing data from volumes in one replica
2022-04-26 22:08:34 -07:00
chrislu
94f824e1ce
volume: sync to disk before copying volume files
...
address https://github.com/chrislusf/seaweedfs/issues/2976
2022-04-26 13:03:43 -07:00
Konstantin Lebedev
7315d1d039
waite volume being closed during compression idx
2022-04-26 13:40:42 +05:00
Konstantin Lebedev
d4343ab7da
forcePurging desc
2022-04-25 23:11:56 +05:00
Konstantin Lebedev
ae56b2c00f
change forcePurging to a pointer
2022-04-25 23:10:01 +05:00
Konstantin Lebedev
6d2fda27d2
delete missing data from volumes in one replica
2022-04-25 22:59:46 +05:00
Konstantin Lebedev
67476d830a
volume.check.disk add param volumeId
2022-04-25 18:27:14 +05:00
chrislu
2aef5b8b32
3.00
2022-04-24 21:46:03 -07:00
Chris Lu
184e982898
Merge pull request #2957 from guo-sj/handle_implicit_username
...
Handle implicit username
2022-04-24 09:07:48 -07:00
guosj
8fd7b24b80
remove redundant logs & add unit test
2022-04-24 16:09:10 +08:00
guol-fnst
8b2552f944
fix http response error code
2022-04-24 11:07:27 +08:00
guosj
09befe871f
Merge branch 'master' into handle_implicit_username
2022-04-23 09:48:25 +08:00
guosj
893a0eea57
add more checks and comments
2022-04-22 22:47:11 +08:00
chrislu
ad01c63b84
conditionally skip hdfs related code
2022-04-21 01:43:01 -07:00
chrislu
4aa39ef33f
conditionally compile sqlite
2022-04-21 01:22:38 -07:00
chrislu
3885374edf
conditionally build elastic, gocdk to reduce binary size
2022-04-21 01:10:46 -07:00
chrislu
6994e88192
fix typo
2022-04-21 00:02:18 -07:00
chrislu
cb1e7aa27d
fix build
2022-04-20 16:21:56 -07:00
a
1d6a9e66b6
Merge branch 'master' into a
2022-04-20 14:01:42 -07:00
chrislu
bdc4d67de8
Revert "Merge pull request #2944 from guo-sj/handle_implicit_username"
...
This reverts commit bc96e73b8d88a669dd19
2022-04-20 09:40:41 -07:00
chrislu
3aa4dc6ad5
Revert "fix index out of range"
...
This reverts commit 4792e07228
2022-04-20 09:40:24 -07:00
chrislu
4792e07228
fix index out of range
2022-04-20 09:32:17 -07:00
chrislu
f3fdfb07b8
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-04-19 22:58:59 -07:00
chrislu
8e9ad7db5a
fix bug deleting volume or unmount volume if a volume server has multiple directories
2022-04-19 22:58:57 -07:00
Chris Lu
bc96e73b8d
Merge pull request #2944 from guo-sj/handle_implicit_username
...
handle implicit username
2022-04-19 21:42:21 -07:00
guosj
a23629b2fe
handle implicit username
2022-04-20 10:12:33 +08:00
Chris Lu
88a669dd19
Merge pull request #2858 from kmlebedev/fsck_replicas
...
fsck replicas
2022-04-19 00:17:43 -07:00
Chris Lu
f76c318a1f
Merge pull request #2940 from kmlebedev/volume.list.grep
...
volume.list show only readonly, collectionPattern and volumeId
2022-04-19 00:17:32 -07:00
Konstantin Lebedev
2364fab927
volume.list show only readonly, collectionPattern and volumeId
2022-04-19 11:44:41 +05:00
guosj
94ad557d97
add missing return
2022-04-19 14:16:32 +08:00
Konstantin Lebedev
36c5a59ed8
add help
2022-04-18 19:36:14 +05:00
Konstantin Lebedev
1e35b4929f
shell vacuum volume by collection and volume id
2022-04-18 18:40:58 +05:00
Chris Lu
97c6e0efb1
Merge pull request #2931 from kmlebedev/fix_nil_dataBackend
...
avoid invalid memory address or nil pointer dereference
2022-04-18 00:13:26 -07:00
Konstantin Lebedev
9438738693
avoid invalid memory address or nil pointer dereference
2022-04-18 12:10:22 +05:00
chrislu
a69f0b62c2
start filer local socket only on non windows
...
fix https://github.com/chrislusf/seaweedfs/issues/2930
2022-04-18 00:08:31 -07:00
chrislu
9ae66f55fc
2.99
2022-04-17 23:01:42 -07:00
chrislu
4524ceae6b
less verbose logs
...
fix https://github.com/chrislusf/seaweedfs/discussions/2925#discussioncomment-2584085
2022-04-17 22:55:26 -07:00
leyou240
89eb87c1d1
Merge branch 'master' into slices.SortFunc
2022-04-18 10:39:29 +08:00
justin
3551ca2fcf
enhancement: replace sort.Slice with slices.SortFunc to reduce reflection
2022-04-18 10:35:43 +08:00
chrislu
b597baf488
mount: set file write time correct to work well with rsync
...
fix https://github.com/chrislusf/seaweedfs/issues/2867
2022-04-16 22:56:45 -07:00
chrislu
460d56d283
shell: cluster.check prints out clock delta and network latency
2022-04-16 13:24:17 -07:00
chrislu
b4be56bb3b
add timing info during ping operation
2022-04-16 12:45:49 -07:00
chrislu
1384529eb7
Fix filer.backup deletes files in backup folder in incremental mode
...
fix https://github.com/chrislusf/seaweedfs/issues/2919
2022-04-14 13:35:01 -07:00
a
846858fb43
merge master
2022-04-12 14:42:29 -07:00
chrislu
1f03fcccb1
fix nil in cluster_check shell command
...
fix https://github.com/chrislusf/seaweedfs/issues/2905
2022-04-12 08:47:27 -07:00
Chris Lu
dd13764a5c
Merge pull request #2900 from kmlebedev/fix_cleanupUploads
...
avoid breaking loop in cleanupUploads if error is empty
2022-04-12 02:03:31 -07:00
Konstantin Lebedev
76b1c5ce67
cleanupUploads use jwt token
2022-04-12 13:46:11 +05:00
Chris Lu
72db181d68
Merge pull request #2901 from guol-fnst/fix_multiupload
...
check object name and uploadID when processing multiupload
2022-04-11 23:38:12 -07:00
guol-fnst
44d810d163
rename functions and remove uncessary check
2022-04-12 13:40:53 +08:00
guol-fnst
180aa88a92
check uploadid using object name hash string
2022-04-12 11:04:38 +08:00
guol-fnst
2232cfb5b7
Check object name and uploadID when processing multipart uploading
2022-04-11 19:53:44 +08:00
Konstantin Lebedev
2dfbe210e0
avoid breaking loop in cleanupUploads if error is empty
2022-04-11 11:16:20 +05:00
Chris Lu
a87f57e47c
Merge pull request #2868 from kmlebedev/hashicorp_raft
...
hashicorp raft
2022-04-10 23:00:05 -07:00
Konstantin Lebedev
931cb9e581
use "cluster.raft.{ps,add,remove}"
2022-04-11 10:50:01 +05:00
Konstantin Lebedev
d2fe0fae33
use the full ServerAddress as the identifier
2022-04-11 10:44:44 +05:00
chrislu
c6ec5269f4
2.98
2022-04-10 18:56:40 -07:00
chrislu
261ba0208e
lock-protected setting entry
2022-04-09 22:52:59 -07:00
Chris Lu
9f4ecd50ac
Merge pull request #2889 from blacktear23/filer-ui-bootstrap-style
...
Change filer UI icon from picture to icon font.
2022-04-08 00:50:32 -07:00
yulai.li
ceb058ddd9
Change filer UI icon from picture to icon font.
2022-04-08 15:07:56 +08:00
chrislu
7f26676f57
adjust td width
2022-04-07 22:34:32 -07:00
yulai.li
dff63e9024
Make operation buttons show when table row hovered
2022-04-08 13:10:12 +08:00
chrislu
4dba102137
use icon instead of button text
2022-04-07 11:15:01 -07:00
Chris Lu
bbc95dac47
Merge pull request #2885 from binbinshi/master
...
fix: master lose some volumes
2022-04-07 10:16:19 -07:00
Konstantin Lebedev
104ea7b029
master ui add raft stats and Max Volume Id
2022-04-07 20:52:01 +05:00
a
41d396edc4
Merge branch 'master' into a
2022-04-07 08:22:59 -07:00
Konstantin Lebedev
a2fdb3e277
fix master ui
2022-04-07 19:37:40 +05:00
Konstantin Lebedev
35bc67f030
fix conflicts
2022-04-07 18:52:13 +05:00
Konstantin Lebedev
f5246b748d
Merge branch 'new_master' into hashicorp_raft
...
# Conflicts:
# weed/pb/master_pb/master.pb.go
2022-04-07 18:50:27 +05:00
yulai.li
11a9f993a8
Make filer-ui bootstrap style
2022-04-07 20:17:00 +08:00
Konstantin Lebedev
7ff248d5cd
refactor OnPeerUpdate
2022-04-07 16:23:22 +05:00
Konstantin Lebedev
85d80fd36d
fix removing old raft server
2022-04-07 15:31:37 +05:00
chrislu
abe3cc6df2
filer UI touch up
2022-04-07 01:25:55 -07:00
Chris Lu
869d23b099
Merge pull request #2884 from blacktear23/filer-ui-new-features
...
Filer UI new features
2022-04-07 00:58:56 -07:00
chrislu
86ed27f602
mount: remove leftover socket file
2022-04-07 00:33:13 -07:00
yulai.li
2347c21cdd
Tune filer UI add rename feature
2022-04-07 15:19:41 +08:00
yulai.li
2454020a92
Add upload progress support
2022-04-07 15:19:37 +08:00
yulai.li
4f87ee7755
Add create directory and delete web UI features for filer
2022-04-07 15:19:33 +08:00
shibinbin
c20e1edd99
fix: master lose some volumes
2022-04-07 15:18:28 +08:00
eddy-gfx
ec53eec94f
Merge branch 'master' into a
2022-04-06 18:45:55 -05:00
chrislu
3ab2c0e5c0
avoid possible nil attributes
2022-04-06 10:11:11 -07:00
chrislu
9d44b89c21
Revert "avoid possible nil attributes"
...
This reverts commit 0a6703c7f7
2022-04-06 10:01:45 -07:00
chrislu
0a6703c7f7
avoid possible nil attributes
2022-04-06 09:58:48 -07:00
Konstantin Lebedev
14a2cc83bf
raft update peers via OnPeerUpdate
2022-04-06 21:17:04 +05:00
Konstantin Lebedev
357aa818fe
add raft shell cmds
2022-04-06 15:23:53 +05:00
chrislu
3d229fe45c
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-04-06 00:28:51 -07:00
chrislu
79b8e6a8c3
add a place holder for later merge consecutive chunks
2022-04-06 00:28:48 -07:00
Konstantin Lebedev
c1aeded2dd
force raft bootstrap
...
avoid err bootstrap only works on new clusters
2022-04-06 12:18:43 +05:00
Chris Lu
49ecb8d11c
Merge pull request #2879 from leyou240/master
...
remove Redundant type conversion and use strings.TrimSuffix to enhance readability
2022-04-06 00:02:19 -07:00
justin
d02f13c2d1
remove Redundant type conversion and use strings.TrimSuffix to enhance readability
2022-04-06 14:58:09 +08:00
root
f15a737a0a
Add "Location:" in response when creating bucket
...
according to "https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html "
2022-04-06 13:56:27 +08:00
chrislu
bc888226fc
erasure coding: tracking encoded/decoded volumes
...
If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.
2022-04-05 19:03:02 -07:00
chrislu
8b3d76b24d
prevent concurrent access during shutdown
2022-04-05 10:53:36 -07:00
chrislu
bb4beebce3
prevent nil
2022-04-05 10:49:17 -07:00
Konstantin Lebedev
0e796a5582
rm set NoSnapshotRestoreOnStart
2022-04-05 18:28:42 +05:00
Konstantin Lebedev
68f11b9687
sleep bootstraping
2022-04-05 15:42:19 +05:00
Konstantin Lebedev
1ae7f509f5
LocalID from raw string
2022-04-05 13:50:39 +05:00
Konstantin Lebedev
89d32a0670
fix glog Info
2022-04-05 13:43:34 +05:00
Konstantin Lebedev
17c6e8e39f
Merge branch 'new_master' into hashicorp_raft
...
# Conflicts:
# go.mod
# go.sum
2022-04-05 13:29:46 +05:00
Konstantin Lebedev
b7cdde14ae
auto bootstraping and update peers
2022-04-05 13:17:53 +05:00
chrislu
4aae87f405
check missing hdd disk type
2022-04-04 14:48:00 -07:00
Konstantin Lebedev
622297f1a7
add stats raft handler
2022-04-04 19:16:06 +05:00
Konstantin Lebedev
1a1e5778c3
fix cluster status
2022-04-04 18:52:08 +05:00
Konstantin Lebedev
14dd971890
hashicorp raft with state machine
2022-04-04 17:51:51 +05:00
Konstantin Lebedev
c514710b7b
initial add hashicorp raft
2022-04-04 13:50:56 +05:00
chrislu
d310711de0
avoid possible deadlock: move metadata log flush channel out of lock scope
2022-04-03 21:53:30 -07:00
chrislu
77a7d7253f
2.97
2022-04-03 19:08:01 -07:00
chrislu
e8d7bb42e2
avoid concurrent access to map object
...
fix https://github.com/chrislusf/seaweedfs/issues/2866
2022-04-03 01:08:25 -07:00
chrislu
6a2bcd03aa
configure mount quota
2022-04-02 21:34:26 -07:00
chrislu
958f880b70
mount: add grpc method to adjust quota
2022-04-02 15:14:37 -07:00
chrislu
a8eefda452
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-04-01 20:25:37 -07:00
chrislu
105578a2f2
skip pinging self for master and volume server
2022-04-01 20:25:35 -07:00
root
b7e34016d4
fix error response in case of wrong expire time format
2022-04-02 11:01:33 +08:00
chrislu
4ecba915f3
add check between peers
2022-04-01 17:40:25 -07:00
chrislu
daca2d22a5
use original server address string as map key
2022-04-01 17:34:42 -07:00
chrislu
4b5c0e3fa9
check cluster connectivities
2022-04-01 17:27:49 -07:00
chrislu
6d55716fc3
use constants
2022-04-01 16:55:26 -07:00
chrislu
bbbbbd70a4
master supports grpc ping
2022-04-01 16:50:58 -07:00
chrislu
743ad690b6
filer supports grpc ping
2022-04-01 16:44:58 -07:00
chrislu
2305508b65
refactor: separate into two files
2022-04-01 16:40:49 -07:00
chrislu
800cbc004c
volume server adds ping function
2022-04-01 16:37:06 -07:00
a
7e92517571
change user and pass to username and password
2022-04-01 14:09:25 -05:00
Konstantin Lebedev
7f1383a41e
findExtraChunksInVolumeServers in consideration of replication
2022-04-01 14:45:41 +05:00
Konstantin Lebedev
3817e05dd0
fix collect filer files
2022-04-01 10:17:09 +05:00
Konstantin Lebedev
3cedb21bb7
skip new entities
2022-03-31 21:36:10 +05:00
Konstantin Lebedev
1d9e30d8c0
fsck replicas
2022-03-31 19:10:06 +05:00
Konstantin Lebedev
4f5b018a02
collecting only bucket volumes
2022-03-31 13:35:58 +05:00
Konstantin Lebedev
691ed50957
use constant for ErrBucketNotEmpty
2022-03-31 08:44:48 +05:00
Konstantin Lebedev
b6a2b43efe
allowDeleteBucketNotEmpty
2022-03-30 22:46:13 +05:00
Konstantin Lebedev
a8b15c676c
s3 test bucket delete nonempty
2022-03-30 21:06:26 +05:00
a
549e341c6f
resolv
2022-03-30 06:31:19 -05:00
Chris Lu
ce8ef60c59
Merge pull request #2840 from kmlebedev/fix_multi_object_delete_key_limit
...
fix test_s3.test_multi_object_delete_key_limit and test_multi_objectv2_delete_key_limit
2022-03-30 01:27:13 -07:00
Konstantin Lebedev
6317f6a8a9
fix Description
2022-03-30 13:04:15 +05:00
Chris Lu
8732cc24c8
Merge pull request #2849 from guo-sj/update_user
...
Add AWS IAM update user API and its test case
2022-03-29 22:36:26 -07:00
Konstantin Lebedev
993283bb1d
revert sort.Search
2022-03-29 19:53:12 +05:00
Konstantin Lebedev
5b90a39954
fix s3 tests:
...
multipart_upload_incorrect_etag
multipart_resend_first_finishes_last
2022-03-29 19:21:09 +05:00
Konstantin Lebedev
9231971c88
fix test_s3.test_multi_object_delete_key_limit and
...
test_multi_objectv2_delete_key_limit
2022-03-29 16:55:09 +05:00
guosj
ed07e76f0e
add test function
2022-03-29 17:21:59 +08:00
a
bc603e534f
diff
2022-03-28 16:50:28 +00:00
Konstantin Lebedev
84b7b83517
fix permission mkdir snapshot
...
avoid open file operation not permitted
2022-03-28 18:41:52 +05:00
Chris Lu
93615b2a49
Merge pull request #2828 from guo-sj/fix-tagging-error
...
Fix delete all "Seaweed-" prefixed tagging error
2022-03-28 00:35:22 -07:00
guosj
dd2b9d93cc
update code according to PR's comment
2022-03-28 15:23:31 +08:00
guosj
02076f01a6
fix tagging error
2022-03-28 14:50:48 +08:00
justin
d51a724101
fix: encode small chunk return error maybe have some bug.
2022-03-28 13:11:24 +08:00
chrislu
0490ee87ef
2.96
2022-03-27 16:11:17 -07:00
chrislu
86d4c52d6e
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-26 13:33:20 -07:00
chrislu
21e0898631
refactor: change masters from a slice to a map
2022-03-26 13:33:17 -07:00
chrislu
4ba7127ab1
refactor
2022-03-26 13:13:19 -07:00
Chris Lu
3153c836a7
Merge pull request #2744 from kmlebedev/issues_2545
...
avoid connect to the old filler address
2022-03-26 13:02:49 -07:00
chrislu
cae998eda1
Revert "Revert "remove duplicated metadata subscription in filer""
...
This reverts commit 34b743c481
2022-03-26 13:00:16 -07:00
chrislu
34b743c481
Revert "remove duplicated metadata subscription in filer"
...
This reverts commit 34742be029https://github.com/chrislusf/seaweedfs/issues/2545
2022-03-26 12:33:45 -07:00
chrislu
fba1cfc2d6
simplify a bit
2022-03-26 10:24:05 -07:00
chrislu
a3411dd9da
refactor
2022-03-26 10:21:26 -07:00
Chris Lu
1b27f96669
Merge pull request #2814 from kmlebedev/fix_remove_deleted_peers
...
Fix remove deleted peers
2022-03-25 11:36:15 -07:00
Konstantin Lebedev
ddd3945c26
fix remove deleted peers of raft server
...
https://github.com/chrislusf/seaweedfs/issues/2804
2022-03-25 15:09:38 +05:00
Konstantin Lebedev
c1450bf9fe
always clear previous log to avoid server is promotable
...
https://github.com/chrislusf/seaweedfs/issues/2804
2022-03-25 13:40:19 +05:00
Chris Lu
89d84e275b
Merge pull request #2759 from kmlebedev/skip_wait_cancelled_request
...
Need to exit waiting if request is was canceled
2022-03-24 12:21:44 -07:00
Konstantin Lebedev
0b790d2bbf
exclude the replication from the concurrentUploadLimitMB
2022-03-24 13:54:42 +05:00
chrislu
9b04f17555
remove dead code
2022-03-23 23:18:32 -07:00
chrislu
0b5faef612
fix
2022-03-23 23:16:54 -07:00
chrislu
0562fceb99
volume: fail fast if too many concurrent requests, to avoid dead lock due to replication.
...
fix https://github.com/chrislusf/seaweedfs/issues/2755
2022-03-23 22:53:58 -07:00
Konstantin Lebedev
849a2c1f68
fix cmdFiler
2022-03-23 14:30:46 +05:00
Konstantin Lebedev
105bcfdbd4
set iam ip from filer
2022-03-23 13:45:51 +05:00
chrislu
e48764be75
s3: multipart upload verifies uploaded parts
2022-03-23 01:05:14 -07:00
chrislu
18ae8943ef
better clean up upload directory
2022-03-23 01:03:51 -07:00
chrislu
a0bad1c997
remove any go mod changes
...
This reverts commit 6c7f7d6887
2022-03-21 23:04:00 -07:00
chrislu
6c7f7d6887
Revert "Merge pull request #2782 from SadmiB/upstream"
...
This reverts commit a644b7236a349257f822
2022-03-21 23:00:50 -07:00
Chris Lu
0d692a6254
Merge branch 'master' into upstream
2022-03-21 11:02:04 -07:00
SadmiB
d12540c9f2
Add contabo api client
2022-03-21 17:16:49 +01:00
chrislu
4e2388e1b5
mount: set file size if it is only on remote gateway
2022-03-21 02:09:54 -07:00
chrislu
8f0410af2c
2.95
2022-03-21 01:47:03 -07:00
chrislu
ae558fa073
log reasons volumes became unwritable
2022-03-21 00:41:44 -07:00
chrislu
57c6eddd22
avoid possible deadlock if volume layout is used in some logs
2022-03-21 00:04:01 -07:00
chrislu
581c93d970
change to config_dir, to be consistent with "weed scaffold -config"
...
fix https://github.com/chrislusf/seaweedfs/issues/2753
2022-03-19 00:24:52 -07:00
chrislu
b75c6dcff5
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-19 00:22:52 -07:00
chrislu
3da2b83b38
Added a "-conf_dir" option to customize *.toml configuration file directory.
...
fix https://github.com/chrislusf/seaweedfs/issues/2753
2022-03-19 00:22:47 -07:00
elee
beb406bbbb
fix ls, onBucketDelete still not triggering
2022-03-18 22:21:57 -05:00
elee
411c0df3fe
switch to multi collection, change readme
2022-03-18 21:51:16 -05:00
zzq09494
9f16df736e
fix-filer: calculation error of the method skipCheckParentDirEntry
2022-03-19 09:15:42 +08:00
elee
1cea6c73d3
update readme
2022-03-18 00:34:19 -05:00
elee
25be96832a
update readme with index info
2022-03-18 00:33:08 -05:00
elee
06f23aa675
put in delete folder children query
2022-03-18 00:29:48 -05:00
elee
0701feeb17
added more to readme
2022-03-18 00:26:11 -05:00
elee
171c27ddf1
zz
2022-03-17 22:35:30 -05:00
elee
2f0cdcdceb
update readme
2022-03-17 21:26:16 -05:00
elee
423ce57cde
prefix search, bucket implemented
2022-03-17 21:12:25 -05:00
chrislu
2824940ecf
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-17 16:54:30 -07:00
chrislu
f247cab5cd
skip localhost if bound to all interfaces already 0.0.0.0 or 127.0.0.1
2022-03-17 16:54:29 -07:00
Berck Nash
7ee38fa3a4
The fixes for https://github.com/chrislusf/seaweedfs/issues/1937 had a few problems:
...
(1) The help file says that in the absence of a ipBind being specified, that it will bind to the "ip" specified. Instead, it bound to localhost which broke the default configuration. This change implements the documented behavior instead.
(2) The new IAM filer ip address has no default. This instantiates it to the same as the filer IP. I'm not sure if there should be a corresponding iam.ip or iam.ipBind option added to the filer command?
2022-03-17 15:30:23 -06:00
elee
bf745bdccb
revise comment
2022-03-17 14:09:34 -05:00
elee
a0df993cef
cleanup, add a few comments
2022-03-17 14:08:36 -05:00
elee
94883e2fad
ok then
2022-03-17 05:23:12 -05:00
elee
f3ab6769e9
make start from indexing work
2022-03-17 05:09:13 -05:00
elee
31571dd96e
fix file listing
2022-03-17 05:07:40 -05:00
elee
b25d03340b
Merge branch 'a' of github.com:gfxlabs/seaweedfs into a
2022-03-17 04:51:59 -05:00
elee
921535001a
arangodb adapter
2022-03-17 04:49:26 -05:00
chrislu
6886167d5f
minor optimization
2022-03-17 00:03:28 -07:00
chrislu
7542fd7f4d
mount: optimize a bit when writing files
2022-03-17 00:02:38 -07:00
chrislu
4042fdf3bb
rename to skipCheckParentDir
...
related to https://github.com/chrislusf/seaweedfs/pull/2761
It's better to default to false.
2022-03-16 23:55:31 -07:00
zzq09494
40b0033fa7
go fmt
2022-03-17 14:19:48 +08:00
zzq09494
ee5417a5b3
fix: ut test error
2022-03-17 11:16:46 +08:00
zzq09494
81cce4b4c3
filer: support uploading file without needEnsureParentDir
2022-03-17 10:53:47 +08:00
zzq09494
a6a8892255
Revert "filer: support uploading file without needEnsureParentDir"
...
This reverts commit a93c4947ba
2022-03-17 10:27:17 +08:00
zzq09494
a93c4947ba
filer: support uploading file without needEnsureParentDir
2022-03-17 10:18:23 +08:00
Berck Nash
9b14f0c81a
Add mTLS support for both master and volume http server.
2022-03-16 09:52:17 -06:00
chrislu
ed818c90a9
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-15 22:28:21 -07:00
chrislu
3639cad69c
master, filer, s3: also listen to "localhost" in addition to specific ip address
...
related to https://github.com/chrislusf/seaweedfs/issues/1937
2022-03-15 22:28:18 -07:00
Konstantin Lebedev
f43c6daeda
Need to exit waiting if request is was canceled
2022-03-15 19:55:22 +05:00
Konstantin Lebedev
c2e7d663bc
reduced the timeout for creating a connection from 30 to 10 seconds
2022-03-15 13:32:05 +05:00
chrislu
fbc9f0eb64
minor
2022-03-14 03:19:16 -07:00
chrislu
2eda3a686f
2.94
2022-03-14 00:55:01 -07:00
chrislu
bd5c5586b5
generate inode via path and time
2022-03-14 00:03:29 -07:00
chrislu
5cba8e51c5
refactor
2022-03-13 18:34:57 -07:00
chrislu
f2f68f675e
write to disk during random writes, limiting total disk spaces used
2022-03-13 18:17:35 -07:00
chrislu
2f4cd20f90
tests skip reader pattern monitoring
2022-03-13 18:15:53 -07:00
chrislu
53513475bf
mount: add back random read support
...
avoid too much memory used also
2022-03-13 01:38:52 -08:00
chrislu
f70c1e449b
add useful doc link
2022-03-13 00:14:50 -08:00
chrislu
b20ddc57a7
mount: return open status
2022-03-12 22:38:14 -08:00
chrislu
b40d252761
mount: chmod for root
2022-03-12 12:10:56 -08:00
chrislu
3a6eb8ca5f
default bind to one ip address
...
fix https://github.com/chrislusf/seaweedfs/issues/1937
2022-03-11 14:02:39 -08:00
Konstantin Lebedev
f53cff045f
checks disk file exist
2022-03-10 18:58:56 +05:00
Konstantin Lebedev
834210a9dc
avoid connect to the old filler address
...
https://github.com/chrislusf/seaweedfs/issues/2545
2022-03-10 15:24:45 +05:00
chrislu
197ade6aeb
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-09 22:26:55 -08:00
chrislu
011a41b561
add back writes to swap file when too many in memory chunks are used.
2022-03-09 22:26:51 -08:00
zzq09494
2ea18fca48
fix:support some databases which not support 'IF NOT EXISTS' keyword of 'create table command'
2022-03-10 13:37:34 +08:00
banjiaojuhao
f28dbbe5c5
[bugfix] filer: 1. Delete uploaded chunks when upload failed. 2. Report error when upload is interrupted by user.
2022-03-10 11:40:39 +08:00
chrislu
e2b07737da
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-08 23:02:33 -08:00
chrislu
dc204dd137
fix nil entry
...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1d340b4]
goroutine 130523 [running]:
github.com/chrislusf/seaweedfs/weed/filer.FileSize(...)
/code/seaweedfs/weed/filer/filechunks.go:26
github.com/chrislusf/seaweedfs/weed/mount.(*WFS).Lookup(0xc000866d80, 0x1, 0xc002897f40, {0xc004b00980, 0x39}, 0x1ec19e0)
/code/seaweedfs/weed/mount/weedfs_dir_lookup.go:59 +0x654
github.com/hanwen/go-fuse/v2/fuse.doLookup(0xc00033c000, 0xc00033c000)
/code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/opcode.go:333 +0x6b
github.com/hanwen/go-fuse/v2/fuse.(*Server).handleRequest(0xc000ab2420, 0xc00033c000)
/code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/server.go:483 +0x1f3
github.com/hanwen/go-fuse/v2/fuse.(*Server).loop(0xc000ab2420, 0x0)
/code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/server.go:456 +0x110
created by github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest
/code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/server.go:323 +0x534
2022-03-08 23:02:30 -08:00
banjiaojuhao
f7f2a597dd
minor
2022-03-08 16:22:55 +08:00
banjiaojuhao
d61bea9038
[bugfix] filer: In file modification, old chunks will be mis-deleted when they are merged(Manifestized).
2022-03-08 16:22:55 +08:00
chrislu
3aeee3d748
ensure releasing file handle
2022-03-07 14:01:24 -08:00
chrislu
8136384473
remove debug message
2022-03-07 11:22:26 -08:00
banjiaojuhao
b9ff7723dd
[bugfix] filer: nil pointer dereference
2022-03-07 23:26:25 +08:00
chrislu
da3d330616
s3 and filer transport using unix domain socket instead of tcp
2022-03-07 02:00:14 -08:00
chrislu
0cb17b45b1
refactoring
2022-03-07 01:59:01 -08:00
chrislu
6d3db4445b
buffer for all range requests
2022-03-07 01:56:47 -08:00
chrislu
f3bcbeb60a
a little optimization
2022-03-07 00:24:59 -08:00
chrislu
f7ee60996c
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-07 00:08:09 -08:00
chrislu
6f3ec989a7
fix manifest batch size
2022-03-07 00:07:53 -08:00
Chris Lu
1e7fcef581
Merge pull request #2729 from banjiaojuhao/filer_metadata-resolve-manifest
...
filer: support get metadata with resolved manifest chunk
2022-03-07 00:05:46 -08:00
chrislu
bb0b784544
minor
2022-03-07 00:04:59 -08:00
banjiaojuhao
bfcc9ca808
filer: support metadata with resolved manifest chunk
2022-03-07 15:47:51 +08:00
banjiaojuhao
71f3046841
filer: add back isAppend function
2022-03-07 15:41:07 +08:00
chrislu
0ba4e4cd23
2.93
2022-03-06 18:54:12 -08:00
chrislu
46a28b8819
mount: adjust disk space based on quota
2022-03-06 17:22:49 -08:00
chrislu
ede6ce44c6
fix test
2022-03-06 17:09:55 -08:00
chrislu
21ef152423
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-03-06 17:04:24 -08:00
chrislu
f3442e36e6
mount: quota adjust error type to be syscall.ENOSPC
2022-03-06 17:04:21 -08:00
Chris Lu
89cd0c27a8
Merge pull request #2728 from kmlebedev/set_default_leveldb2
2022-03-06 07:26:30 -08:00
Konstantin Lebedev
cf444ebd07
Set default leveldb2 enabled
...
avoid Filer store is enabled for both leveldb2 and mysql
2022-03-06 18:27:25 +05:00
banjiaojuhao
a07c93f4dd
filer: reset bytesBuffer before use
2022-03-06 21:07:36 +08:00
chrislu
c7e8ac18f0
mount: quota for one mounted collection
...
related to https://github.com/seaweedfs/seaweedfs-csi-driver/issues/48
2022-03-06 02:44:40 -08:00
chrislu
b7c992f410
add flag to enforce quota
2022-03-05 22:10:43 -08:00
chrislu
f1713c96ae
avoid possible runtime error: index out of range [0] with length 0
2022-03-05 21:14:31 -08:00
chrislu
da76af187f
mount: avoid possible index out of bounds error
2022-03-04 22:36:01 -08:00
chrislu
6e49e75a5b
use logical number of files and sizes for statistics and quota
2022-03-04 18:47:44 -08:00
chrislu
f51e20028a
mount: avoid comma in mount options
...
fix https://github.com/chrislusf/seaweedfs/issues/2719
2022-03-03 03:42:29 -08:00
chrislu
dc0f48682c
unused
2022-03-03 03:41:35 -08:00
chrislu
28b8974a3a
mount: fix directory pagination when using midnight commander
2022-03-03 02:59:31 -08:00
chrislu
a96d4254e9
filer, s3, volume server: a bit memory optimization
2022-03-02 20:15:28 -08:00
chrislu
6fbbc78574
stream reading a whole chunk
2022-03-02 13:50:46 -08:00
chrislu
784583afc6
avoid pool memory allocation if too large
2022-03-02 13:50:28 -08:00
chrislu
ba14307319
2.92
2022-02-28 15:22:19 -08:00
chrislu
fcf3714443
mount: add back support for filer.path
2022-02-28 12:16:53 -08:00
chrislu
80c017907b
filer.backup: fix backing up encrypted chunks
...
I have done filer.backup test:
replication.toml:
[sink.local]
enabled = true
directory = "/srv/test"
___
system@dat1:/srv/test$ weed filer.backup -filer=app1:8888 -filerProxy
I0228 12:39:28 19571 filer_replication.go:129] Configure sink to local
I0228 12:39:28 19571 filer_backup.go:98] resuming from 2022-02-28 12:04:20.210984693 +0100 CET
I0228 12:39:29 19571 filer_backup.go:113] backup app1:8888 progressed to 2022-02-28 12:04:20.211726749 +0100 CET 0.33/sec
system@dat1:/srv/test$ ls -l
total 16
drwxr-xr-x 2 system system 4096 Feb 28 12:39 a
-rw-r--r-- 1 system system 48 Feb 28 12:39 fu.txt
-rw-r--r-- 1 system system 32 Feb 28 12:39 _index.html
-rw-r--r-- 1 system system 68 Feb 28 12:39 index.php
system@dat1:/srv/test$ cat fu.txt
? ?=?^??`?f^};?{4?Z%?X0=??rV????|"?1??踪~??
system@dat1:/srv/test$
On the active mount on the target server it's:
system@app1:/srv/app$ ls -l
total 2
drwxrwxr-x 1 system system 0 Feb 28 12:04 a
-rw-r--r-- 1 system system 20 Feb 28 12:04 fu.txt
-rw-r--r-- 1 system system 4 Feb 28 12:04 _index.html
-rw-r--r-- 1 system system 40 Feb 28 12:04 index.php
system@app1:/srv/app$ cat fu.txt
This is static boy!
Filer was started with: weed filer master="app1:9333,app2:9333,app3:9333" -encryptVolumeData
It seems like it's still encrypted?
2022-02-28 10:07:06 -08:00
chrislu
aad62ee148
mount: mark directory uncached if forgotten
2022-02-28 02:08:24 -08:00
chrislu
554e239097
release file handle
2022-02-28 01:23:14 -08:00
chrislu
3639fedd01
mount: fix fsync opened and renamed files
2022-02-28 00:34:17 -08:00
chrislu
63a9d8f01d
ensure inodes are not duplicating unless hardlinked
2022-02-27 23:13:49 -08:00
chrislu
de77d00c81
correctly clean up for a file
2022-02-27 23:12:28 -08:00
chrislu
18543c6e8b
minor
2022-02-27 23:11:09 -08:00
chrislu
09cd00f356
2.91
2022-02-27 04:03:39 -08:00
chrislu
941ced60a4
download 2 chunks if at the beginning of a file
2022-02-27 03:57:24 -08:00
chrislu
f9d9eed0c9
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-02-27 03:41:35 -08:00
chrislu
d602d68fd1
remove dead code
2022-02-27 03:41:32 -08:00
Chris Lu
6c07af6014
Merge pull request #2715 from gfx-labs/acl-pr
2022-02-27 03:27:32 -08:00
chrislu
c3792c8352
remove dead code
2022-02-27 03:03:19 -08:00
chrislu
aa9eef81e6
retire mount v1
2022-02-27 02:57:27 -08:00
eddy-gfx
fb940dd807
Merge branch 'chrislusf:master' into acl-pr
2022-02-27 04:52:08 -06:00
elee
881a0fe806
ensure compatibility
2022-02-27 04:50:59 -06:00
elee
954ad98e0d
set canned acl on replication create
2022-02-27 04:49:31 -06:00
chrislu
6e1ab97988
use debug option to see operations
2022-02-27 02:02:30 -08:00
chrislu
2112d99140
mount2: add back readonly mode
2022-02-27 01:13:32 -08:00
chrislu
9ef5bb20f6
mount2: invalidate fuse cache for replaced inode
2022-02-27 00:00:23 -08:00
chrislu
4ddcbaab57
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-02-26 23:20:47 -08:00
chrislu
551d00d51a
prefetch other chunks when stream reading
2022-02-26 23:20:45 -08:00
Chris Lu
e1d3dd5e18
Merge pull request #2712 from guo-sj/correct_comments
2022-02-26 09:16:12 -08:00
guosj
82cad5e330
correct comments
2022-02-26 22:44:26 +08:00
chrislu
7b1a713d2a
remove dead code
2022-02-26 03:23:15 -08:00
chrislu
3a58b7bac6
a little safer
2022-02-26 03:23:06 -08:00
chrislu
708e14fcfa
avoid possible too big memory allocation
2022-02-26 03:22:41 -08:00
chrislu
3345a50d9b
prefetch 2 chunks
2022-02-26 03:06:17 -08:00
chrislu
86ce69f709
remove logs
2022-02-26 03:00:20 -08:00
chrislu
b2a148cb4c
use file size as max range
2022-02-26 03:00:08 -08:00
chrislu
2ab0ad24a3
use memory pool
2022-02-26 02:59:19 -08:00
chrislu
28b395bef4
better control for reader caching
2022-02-26 02:16:47 -08:00
chrislu
3ad5fa6f6f
chunk cache adds function ReadChunkAt
2022-02-25 21:55:04 -08:00
chrislu
fc7a4957ea
fix mount2 options
2022-02-25 21:22:44 -08:00
chrislu
72c0233938
less logs
2022-02-25 15:34:24 -08:00
chrislu
101e6d80b0
mount2: listXattr return ok if xattr is empty
2022-02-25 14:38:56 -08:00
chrislu
03466f955e
rename: delete source entry metadata only, skipping hard links
2022-02-25 02:57:54 -08:00
chrislu
8080fe4cc1
logs
2022-02-25 02:56:23 -08:00
chrislu
e423548673
rename: pass along entry metadata
2022-02-25 02:53:37 -08:00
chrislu
e8110bb54c
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-02-25 01:17:28 -08:00
chrislu
202a29d014
refactoring
2022-02-25 01:17:26 -08:00
chrislu
91d6785cf3
define metadata action types
2022-02-25 00:54:16 -08:00
chrislu
be3fc77391
mount2: use consistent inode
2022-02-25 00:53:27 -08:00
Chris Lu
9873bae115
Merge pull request #2706 from guo-sj/fix_log_info
2022-02-24 22:48:19 -08:00
guosj
121b31f750
fix incorrect log information
2022-02-25 13:41:20 +08:00
chrislu
ceaf993a27
mount2: add rdev
2022-02-24 14:51:25 -08:00
chrislu
95717d1006
simpler output
2022-02-24 13:50:08 -08:00
chrislu
b05962b90e
rename: handle hard links
2022-02-24 02:59:00 -08:00
chrislu
e31ec04f4f
mount2: POSIX deleted opened file nlink should be 0
2022-02-24 01:59:37 -08:00
chrislu
419e355e9e
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-02-24 01:31:07 -08:00
chrislu
91f0481f4e
mount2: SetAttr set mode correctly
2022-02-24 01:31:04 -08:00
Chris Lu
97a4b66df7
Merge pull request #2704 from guo-sj/fix_bugs_in_return_value
...
fix return value in storage/volume_vacuum.go:444
2022-02-24 00:49:29 -08:00
chrislu
abe7214c1f
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-02-24 00:37:59 -08:00
chrislu
b93d57da31
mount2: dir read opened file
2022-02-24 00:37:57 -08:00
guosj
3e7aa1caf5
fix return value in storage/volume_vacuum.go:444
2022-02-24 15:54:36 +08:00
Chris Lu
6834df77a0
Merge pull request #2701 from guo-sj/fix_bugs_in_return_value
2022-02-23 19:23:37 -08:00
guosj
26f3ab8d4b
fix a return bug in func (c *commandVacuum) Do
2022-02-24 09:04:38 +08:00
chrislu
320637dc7a
use "mv.from" for moving files
2022-02-23 15:34:42 -08:00
chrislu
c29bc9a367
fix error handling
2022-02-23 15:34:25 -08:00
Chris Lu
da58c748bc
Merge pull request #2698 from guo-sj/fix_bugs_in_return_value
2022-02-23 00:41:30 -08:00
Chris Lu
bd092d8318
Merge pull request #2697 from guo-sj/fix_bugs_in_return_value
2022-02-23 00:30:52 -08:00
guosj
d68c27f82d
fix another return value bug
2022-02-23 16:21:25 +08:00
guosj
8f9aa0cddd
fix bugs in return value
2022-02-23 16:17:48 +08:00
Tuan Vuong
d2ec62656d
initialize master address in iam options
2022-02-23 12:01:54 +07:00
banjiaojuhao
6ab09e9071
filer_http: support uploading file with offset
2022-02-22 00:15:00 +08:00
chrislu
497ebbbd45
2.90
2022-02-20 22:00:13 -08:00
chrislu
56da3494cb
leveldb3: adjust memory allocation for each bucket
2022-02-20 13:40:13 -08:00
banjiaojuhao
e6126cef62
filer_web: support moving entry
2022-02-20 23:56:23 +08:00
chrislu
7bc67399e4
listing for filer stores without prefixed query: break if no more progress
2022-02-19 10:12:51 -08:00
chrislu
248c0c8087
minor
2022-02-19 09:54:09 -08:00
chrislu
4ee0a6f47b
filer store: reduce one possible listing operation
2022-02-19 00:43:42 -08:00
chrislu
b3594278c9
optimize a bit
2022-02-18 23:36:10 -08:00
chrislu
61811dc2f1
comments
2022-02-18 22:14:40 -08:00
chrislu
daa27b2119
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-02-18 20:55:10 -08:00
chrislu
9014d00fd0
Revert "s3: listObjectParts return ErrNoSuchUpload if does not exist"
...
This reverts commit 6cf2e7d493
2022-02-18 20:54:54 -08:00
banjiaojuhao
4c30934cd9
filer: support get file entry
2022-02-18 22:52:26 +08:00
chrislu
63062ed7f0
mount2: fix unlink
2022-02-18 01:10:53 -08:00
chrislu
93e12d5f30
fix build
2022-02-18 00:48:00 -08:00
chrislu
d62370d4e0
add todo
2022-02-18 00:47:15 -08:00
chrislu
e8ce30fdc5
mount2: adjust file mode
2022-02-18 00:47:02 -08:00
chrislu
f9d33f70b0
return fuse.Status when looking up by inode
2022-02-18 00:45:43 -08:00
chrislu
b9cf4f12fc
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-02-16 21:33:51 -08:00
chrislu
49b84b6e2a
list entries while reading from remote
2022-02-16 21:32:15 -08:00
Chris Lu
38dcaaa76e
Merge pull request #2683 from guo-sj/fix_fsconfigure_bug
...
fix minor bug in commandFsConfigure.Do
2022-02-16 18:16:47 -08:00
guosj
ca121ecd66
fix minor bug in commandFsConfigure.Do
2022-02-17 09:57:52 +08:00
chrislu
65a19e3abc
fix listing with correct inode
2022-02-16 17:01:39 -08:00
chrislu
6ac066d1dc
count lookup or not
2022-02-16 16:49:03 -08:00
chrislu
a129bda7d9
sync data first before stopping
2022-02-16 09:11:34 -08:00
chrislu
118d0e01a8
less logs
2022-02-16 08:45:07 -08:00
chrislu
a6bc67c34c
less logs
2022-02-16 08:38:51 -08:00
chrislu
2facd65998
fix second listing
2022-02-16 08:16:27 -08:00
chrislu
1013fc90d8
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-02-16 07:57:11 -08:00
chrislu
22739c653a
clean up
2022-02-16 07:57:08 -08:00
Chris Lu
3cb19cf6db
Merge pull request #2680 from kmlebedev/volume_healthz
2022-02-16 04:43:44 -08:00
Konstantin Lebedev
9ea09cc41c
healthz check to avoid drain pod with last replicas
2022-02-16 14:18:36 +05:00
chrislu
65bfeafb5a
add back mkdir
2022-02-16 01:09:21 -08:00
chrislu
37e8fce841
clean up cache on exit
2022-02-16 00:39:21 -08:00
chrislu
1560ec7e26
remove unused code
2022-02-16 00:37:24 -08:00
chrislu
3cbce878f2
mount2: fix directory pagination
2022-02-15 22:42:10 -08:00
chrislu
df51e0c042
mongodb: remove ErrNoDocuments checking since it is only for creating single record, not for querying
2022-02-15 16:18:36 -08:00
chrislu
98cce6a150
mongodb: ensure closing cursor
2022-02-15 16:15:44 -08:00
Chris Lu
aa7fc299b8
Merge pull request #2676 from banjiaojuhao/add_filer_store-redis_lua
...
FilerStore: add redis_lua
2022-02-15 12:10:34 -08:00
banjiaojuhao
fc3b75f2f8
weed/operation/delete_content.go: nil pointer dereference
2022-02-15 23:07:03 +08:00
banjiaojuhao
b5ec346700
FilerStore: add redis_lua
2022-02-15 20:54:57 +08:00
chrislu
17ac5244c3
mount2: avoid double listing directories
2022-02-15 01:44:17 -08:00
guosj
db7766323e
complete project code, remain test code
2022-02-15 17:35:33 +08:00
chrislu
222798d926
mount2: fix for read dir plus on linux
2022-02-15 00:29:17 -08:00
chrislu
4e72863ba5
mount2 add debug mode
2022-02-15 00:26:30 -08:00
Chris Lu
64afbc5235
Merge pull request #2640 from Radtoo/fsck_with_delete
...
Added basic deletion capabilities to fsck.
2022-02-14 23:00:01 -08:00
chrislu
4e181db21a
mount: default disable cache
...
* Prevent cases as https://github.com/seaweedfs/seaweedfs-csi-driver/issues/43
* Improve read write benchmarks
* Improve AI training performance. Most of the files are just read once.
2022-02-14 20:42:33 -08:00
chrislu
ebc22625b4
less noisy on mac
2022-02-14 16:38:01 -08:00
chrislu
377bf31445
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-02-14 13:48:51 -08:00
chrislu
e8420aaed7
fix building for windows freebsd
2022-02-14 13:48:48 -08:00
Chris Lu
d3ee621fce
Merge pull request #2661 from garenchan/ck-dev1
2022-02-14 10:08:37 -08:00
Konstantin Lebedev
526094d2da
StopTimeout 30 sec
2022-02-14 21:42:27 +05:00
Konstantin Lebedev
275e9a4e86
reduce to default http server KillTimeout and StopTimeout
2022-02-14 21:38:24 +05:00
garenchan
bd032eabe7
[UPDATE] Make heartbeat interval and election timeout of masters configurable.
2022-02-14 21:09:07 +08:00
chrislu
ff666104c4
fix GOOS
2022-02-14 03:14:05 -08:00
chrislu
05724a68d4
skip other OS
2022-02-14 02:59:51 -08:00
chrislu
4244ef8b72
fix building on linux
2022-02-14 02:59:27 -08:00
chrislu
b9c2bff931
clean up
2022-02-14 02:14:26 -08:00
chrislu
a990cd29cd
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-02-14 01:52:25 -08:00
chrislu
6a40fd1c65
2.89
2022-02-14 01:52:16 -08:00
Chris Lu
305418fac5
Merge pull request #2667 from kmlebedev/syncReplicatedWrite
...
atomically write to replicas
2022-02-14 01:48:21 -08:00
Chris Lu
aa13168b4d
Merge pull request #2668 from chrislusf/mount2
...
Mount2
2022-02-14 01:41:09 -08:00
chrislu
fe57a2e770
file set attribute
2022-02-14 01:36:10 -08:00
Konstantin Lebedev
0ed76a0556
clearly
2022-02-14 14:10:06 +05:00
chrislu
dbeeda8123
listen for metadata updates
2022-02-14 01:09:31 -08:00
Konstantin Lebedev
36013f63ed
https://github.com/chrislusf/seaweedfs/issues/2648
2022-02-14 13:59:12 +05:00
chrislu
7286e525ad
support write
2022-02-13 23:27:11 -08:00
chrislu
2b955c1713
support read
2022-02-13 22:50:44 -08:00
chrislu
f3c1e00521
rename
2022-02-13 19:16:56 -08:00
chrislu
bb9919b07a
add open release, refactor
2022-02-13 19:14:34 -08:00
Eng Zer Jun
b92df1654c
test: use T.TempDir to create temporary test directory
...
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-02-14 10:38:13 +08:00
chrislu
072f923a9b
add doc
2022-02-13 16:56:35 -08:00
chrislu
3d0e9e5197
supports renaming
2022-02-13 16:34:57 -08:00
chrislu
417b59b893
doc
2022-02-13 14:37:47 -08:00
chrislu
24290fed9d
add comments
2022-02-13 06:02:21 -08:00
chrislu
6200b6abb1
avoid fatal message in some edge cases
2022-02-13 05:59:10 -08:00
chrislu
6a921e15f3
forget() factor in nlookup
2022-02-13 05:49:29 -08:00
chrislu
f8af0f93d9
support link
2022-02-13 05:29:43 -08:00
chrislu
0381338d40
fix import
2022-02-13 04:23:06 -08:00
chrislu
a1ef0e48a9
doc
2022-02-13 04:22:02 -08:00
chrislu
be40ff6048
add symlink
2022-02-13 03:50:16 -08:00
chrislu
3d93570979
supports forget
2022-02-13 03:31:47 -08:00
chrislu
6a42cb6b0b
supports mknod, unlink
2022-02-13 03:09:24 -08:00
chrislu
813b868b9a
add rmdir
2022-02-13 01:43:11 -08:00
chrislu
e85ca10a1a
add mkdir
2022-02-13 01:34:19 -08:00
chrislu
21046c6a28
split files
2022-02-13 01:05:30 -08:00
chrislu
a4c9223b9d
support xattr
2022-02-13 00:58:46 -08:00
chrislu
c81833a192
add directory setAttr
2022-02-12 23:08:56 -08:00
chrislu
5c48c23235
remove println
2022-02-12 22:45:07 -08:00
chrislu
7cfbf1e85f
fix ok status
2022-02-12 22:41:45 -08:00
chrislu
4c75fd5f9c
sync format
2022-02-12 22:41:29 -08:00
chrislu
b0a5193e32
working
2022-02-12 22:21:30 -08:00
chrislu
661a34e23d
Merge branch 'master' into mount2
2022-02-12 17:58:32 -08:00
chrislu
5e814afe88
blocks count
2022-02-12 17:58:12 -08:00
Radtoo
724ce04b1c
We picked the second flag's name.
2022-02-13 00:53:35 +01:00
chrislu
5a0a709016
it runs, but directory listing output is not showing up
2022-02-12 05:27:16 -08:00
chrislu
866981d8ac
rename
2022-02-12 02:49:15 -08:00
chrislu
72faae91e1
implement read directory and read directory plus
2022-02-12 02:48:44 -08:00
chrislu
a10c28ba82
simplify
2022-02-12 01:59:36 -08:00
chrislu
f4d88862c4
can attr root directory
2022-02-12 01:54:16 -08:00
chrislu
180445f5a8
change to use fuse file system
2022-02-11 21:35:09 -08:00
chrislu
45a0fda9bd
need to follow https://github.com/hanwen/go-fuse/blob/master/fuse/api.go
2022-02-11 03:12:52 -08:00
chrislu
f87da798a4
to be re-written following fuse virtual file system
2022-02-11 03:09:30 -08:00
chrislu
9a913457dd
supports stats
2022-02-10 23:23:47 -08:00
chrislu
b6143de52a
mount with name
2022-02-10 22:43:55 -08:00
chrislu
7a0c35674c
clean up previously mounted folder
2022-02-10 20:46:53 -08:00
chrislu
c3f9d9fa2e
initial setup
2022-02-10 20:32:13 -08:00
Radtoo
fbe7ed7927
Revert the the flag to "reallyDeleteFromVolume".
2022-02-10 20:01:48 +01:00
chrislu
21aaa4c1f1
ec.encode: calculate free ec slots based on (maxVolumeCount-volumeCount)
...
fix https://github.com/chrislusf/seaweedfs/issues/2642
2022-02-08 01:51:13 -08:00
chrislu
13f6ec1c4e
test checking ec distribution
2022-02-08 01:50:05 -08:00
chrislu
f34c2ff7c5
use embed txt
2022-02-08 01:14:27 -08:00
chrislu
c8c7c10c3f
volume.tier.move: avoid double counting
...
related to https://github.com/chrislusf/seaweedfs/issues/2637
2022-02-08 00:57:35 -08:00
chrislu
f18803424a
volume.balance: add delay during tight loop
...
fix https://github.com/chrislusf/seaweedfs/issues/2637
2022-02-08 00:53:55 -08:00
chrislu
9860405974
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-02-07 23:08:57 -08:00
chrislu
85c1615b43
filer read empty file may cause OOM in some cases
...
fix https://github.com/chrislusf/seaweedfs/issues/2641
2022-02-07 23:08:54 -08:00
Chris Lu
a2ac540ecc
Merge pull request #2645 from guol-fnst/fix_Precedence
...
fix preconditions
2022-02-07 18:32:16 -08:00
root
7f0c793083
fix preconditions according to https://tools.ietf.org/id/draft-ietf-httpbis-p4-conditional-26.html#preconditions
2022-02-08 10:13:19 +08:00
chrislu
b1cff07ab0
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-02-07 11:38:01 -08:00
chrislu
81f86c381d
s3: avoid nil response
...
fix https://github.com/chrislusf/seaweedfs/issues/2636
2022-02-07 11:37:59 -08:00
Chris Lu
f46763d74f
Merge pull request #2644 from kmlebedev/vacum_fsync
...
updated needle with fsync
2022-02-07 11:33:58 -08:00
Konstantin Lebedev
ef541972f8
updated needle with fsync
2022-02-08 00:10:53 +05:00
Chris Lu
1b7fb3ce84
Merge pull request #2643 from kmlebedev/graceful_volume_stop
2022-02-07 09:58:26 -08:00
Konstantin Lebedev
fb97e234c9
skips compact if store is stopping
2022-02-07 20:16:15 +05:00
chrislu
9405eaefdb
filer.sync: fix replicating partially updated file
...
Run two servers with volumes and fillers:
server -dir=Server1alpha -master.port=11000 -filer -filer.port=11001 -volume.port=11002
server -dir=Server1sigma -master.port=11006 -filer -filer.port=11007 -volume.port=11008
Run Active-Passive filler.sync:
filer.sync -a localhost:11007 -b localhost:11001 -isActivePassive
Upload file to 11007 port:
curl -F file=@/Desktop/9.xml "http://localhost:11007/testFacebook/ "
If we request a file on two servers now, everything will be correct, even if we add data to the file and upload it again:
curl "http://localhost:11007/testFacebook/9.xml "
EQUALS
curl "http://localhost:11001/testFacebook/9.xml "
However, if we change the already existing data in the file (for example, we change the first line in the file, reducing its length), then this file on the second server will not be valid and will not be equivalent to the first file
Снимок экрана 2022-02-07 в 14 21 11
This problem occurs on line 202 in the filer_sink.go file. In particular, this is due to incorrect mapping of chunk names in the DoMinusChunks function. The names of deletedChunks do not match the chunks of existingEntry.Chunks, since the first chunks come from another server and have a different addressing (name) compared to the addressing on the server where the file is being overwritten.
Deleted chunks are not actually deleted on the server to which the file is replicated.
2022-02-07 03:46:28 -08:00
Radtoo
fbb14e0ea8
Adding separate toggle to purge absent vols
...
While this toggle is basically required to clean out entries for deleted volumes, having a separate description + toggling this separately seems like a good idea so people get a chance to check if their volumes are all mounted/connected as expected.
Also renamed forcePurge to just purge.
2022-02-06 23:46:52 +01:00
Radtoo
6b17f45da2
Also delete paths for which a volume is entirely absent.
2022-02-06 23:22:04 +01:00
Radtoo
fa0cfdfb7f
Added basic deletion capabilities to fsck.
2022-02-06 17:05:58 +01:00
chrislu
433fde4b18
move error to a separate file
...
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
2022-02-04 22:57:51 -08:00
Chris Lu
a23fcb9a7c
Merge pull request #2634 from kmlebedev/errorMetrics
...
error metrics for filer and store
2022-02-04 22:35:13 -08:00
Chris Lu
247bbabda5
Merge pull request #2632 from lapshin-vitaly/s3api_errors
...
add s3api error for copy in file, not directory
2022-02-04 22:34:34 -08:00
chrislu
ced3b89395
add util package
2022-02-04 21:34:58 -08:00
chrislu
affe3c2c12
change to util.WriteFile
2022-02-04 21:32:27 -08:00
chrislu
76e297d64f
sync call to write file, avoid vif loading error
...
fix https://github.com/chrislusf/seaweedfs/issues/2633
2022-02-04 11:14:04 -08:00
zerospiel
f3364fec99
weed/s3api: rearrange s3 methods handlers to ensure correct methods requesting
...
Otherwise current calls for some methods (i.e. GetObjectAcl) ends up with wrong method selection (i.e. GetObject).
Added generic comment rule of traversing methods
2022-02-04 15:14:48 +03:00
Konstantin Lebedev
9978f54acf
fix metric names
2022-02-04 16:45:16 +05:00
Konstantin Lebedev
3f4e17aa24
error metrics for filer and store
2022-02-04 14:07:14 +05:00
Lapshinn Vitaly
6bdc274d4d
add s3api error for copy in file, not directory
2022-02-04 03:28:37 +03:00
chrislu
6cf2e7d493
s3: listObjectParts return ErrNoSuchUpload if does not exist
...
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api abort-multipart-upload --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api list-parts --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
{
"Initiator": null,
"Owner": null,
"StorageClass": "STANDARD"
}
If we abort a multipart upload, it appears that records are left behind. We should get a 404 NoSuchKey error.
2022-02-03 12:34:16 -08:00
zerospiel
b54a65ba5a
weed/s3api: added new bucket handlers for more compatibility with AWS S3
...
Protocol
Otherwise any requests to the underlying handlers results in calls to
ListObjects (v1) that may intensively load gateway and volume servers.
Added the following handlers with default responses:
- GetBucketLocation
- GetBucketRequestPayment
Added the following handlers with NotFound and NotImplemented responses:
- PutBucketAcl
- GetBucketPolicy
- PutBucketPolicy
- DeleteBucketPolicy
- GetBucketCors
- PutBucketCors
- DeleteBucketCors
2022-02-03 17:17:05 +03:00
chrislu
6bee1e9714
[volume.check.disk] fix wrong logic to compare
...
fix https://github.com/chrislusf/seaweedfs/issues/2626
minuend - subtrahend
2022-02-01 07:48:28 -08:00
chrislu
7270067289
2.88
2022-01-30 20:25:26 -08:00
chrislu
84c9bc4389
edge case: old entry was not replicated to remote storage
2022-01-30 20:23:24 -08:00
chrislu
b8490fe427
adjust volume count even when not applying the changes
2022-01-28 19:11:46 -08:00
Chris Lu
b3f0f170b4
Merge pull request #2618 from divanikus/master
2022-01-28 03:34:52 -08:00
chrislu
b9b684194f
remove max connection age
...
following https://github.com/grpc/grpc-go/issues/3170#issuecomment-552517779
2022-01-27 02:28:22 -08:00
divanikus
67e3fe996a
async volumeTierMove
2022-01-26 18:22:31 +03:00
chrislu
62d815d1ca
use limited in memory buffer instead of swap file
2022-01-26 00:36:19 -08:00
Chris Lu
9596fce562
Merge pull request #2613 from kmlebedev/masterPromMetrics
...
Master prom metrics
2022-01-25 08:13:14 -08:00
Konstantin Lebedev
5c9259fa3c
fix metrics master name
2022-01-25 14:42:47 +05:00
chrislu
7328962009
revert "POSIX: should not delete if a directory is not empty"
...
revert 0c75f150620c75f15062
2022-01-24 14:21:50 -08:00
Konstantin Lebedev
c9952759c4
metrics master is leader
2022-01-24 20:13:07 +05:00
Konstantin Lebedev
28efe31524
new master metrics
2022-01-24 19:09:43 +05:00
chrislu
3bba2124ef
use a sliding window of in-memory writable chunks
2022-01-23 23:02:05 -08:00
chrislu
520591e6ea
reset swap file chunk after uploading
2022-01-23 18:30:53 -08:00
chrislu
e185d90d24
2.87
2022-01-23 16:18:55 -08:00
chrislu
4a311c7f5e
dedup local metadata subscribers
...
fix https://github.com/chrislusf/seaweedfs/discussions/2542
2022-01-23 16:14:22 -08:00
chrislu
bb6854b972
adjust log
2022-01-22 12:35:09 -08:00
chrislu
2bdd737971
rename file
2022-01-22 08:28:35 -08:00
chrislu
643bbbeb49
rename
2022-01-22 08:27:40 -08:00
chrislu
18ed06b420
use memory when under 16 chunks
2022-01-22 08:11:01 -08:00
chrislu
8e80f3cd65
move upload pipeline locking to a different file
2022-01-22 08:09:55 -08:00
chrislu
dde34fa99d
rename
2022-01-22 08:08:01 -08:00
chrislu
a2aa542370
rename
2022-01-22 08:07:11 -08:00
chrislu
c376ccc5a5
swap file based random write large file upload
2022-01-22 08:06:53 -08:00
chrislu
b9ae16fbc5
fix memory allocation
2022-01-22 08:05:04 -08:00
chrislu
1ee828b768
refactor
...
do not expose internal offset
2022-01-22 06:34:29 -08:00
chrislu
02d0c12cdd
rename
2022-01-22 06:00:10 -08:00
chrislu
5dea5c0449
refactor
2022-01-22 05:59:07 -08:00
chrislu
8aa6bf0bb9
refactoring
2022-01-22 05:40:10 -08:00
chrislu
d97bd54e63
just refactoring
2022-01-22 04:18:54 -08:00
chrislu
482014f9da
rename file
2022-01-22 03:50:18 -08:00
chrislu
3b4a9addaf
rename
2022-01-22 01:46:10 -08:00
chrislu
4acfc098e9
re-order
2022-01-22 01:43:14 -08:00
chrislu
e71dcfb3a6
add logging for memory allocation
2022-01-22 01:35:12 -08:00
chrislu
9d0f58c329
skip printing fs configuration
2022-01-21 13:29:47 -08:00
chrislu
ce2049cdb6
refactoring, move genFn before saveFn
2022-01-21 12:08:58 -08:00
chrislu
e47f63d159
enforce bucket quota
2022-01-21 02:34:42 -08:00
chrislu
6e57d8d0de
s3: check bucket usage and adjust read only according to quota
2022-01-21 02:15:27 -08:00
chrislu
606667f205
able to configure the quota for a bucket
2022-01-21 01:42:20 -08:00
chrislu
b1063162b6
display bucket quota
2022-01-21 00:55:04 -08:00
chrislu
f103491912
s3: list bucket size from weed shell
2022-01-21 00:26:49 -08:00
chrislu
6c7135d77e
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-01-21 00:17:06 -08:00
chrislu
ea57654e34
refactoring
2022-01-21 00:16:50 -08:00
Chris Lu
7c66f3b5fb
Merge pull request #2602 from kmlebedev/master_metrics
...
master metricsHttpPort
2022-01-20 09:26:25 -08:00
chrislu
b3e526ba95
url should be always using forward slash
2022-01-19 22:16:26 -08:00
Konstantin Lebedev
77c98b657e
master metricsHttpPort
2022-01-19 21:43:22 +05:00
chrislu
77362700e1
S3: fail fast when "X-Amz-Copy-Source" is a folder
...
fix #2593
2022-01-18 12:04:40 -08:00
chrislu
05c3c3f56b
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-01-17 23:38:13 -08:00
chrislu
9b77f0054e
2.86
2022-01-17 23:38:03 -08:00
Chris Lu
ec254d8a89
Merge pull request #2597 from guol-fnst/gocql_to
...
add gocql timeout setting
2022-01-17 23:35:13 -08:00
chrislu
9274557552
keep dirty pages based on temp file
2022-01-17 23:23:49 -08:00
guol-fnst
da9540e666
add gocql timeout setting
2022-01-18 15:21:13 +08:00
chrislu
c87b8f4c30
S3: fail fast when "X-Amz-Copy-Source" is a folder
...
fix https://github.com/chrislusf/seaweedfs/issues/2593
2022-01-17 23:09:37 -08:00
chrislu
b2acfd75e9
ensure entry view cache is invalidated
2022-01-17 23:02:30 -08:00
chrislu
f4ad63528a
wait for reading threads to complete before dropping sealed chunks
2022-01-17 22:24:44 -08:00
chrislu
0a3f95ca01
more logs
2022-01-17 20:41:00 -08:00
chrislu
b068bc291d
testing with always resetting entry view cache
2022-01-17 20:07:01 -08:00
chrislu
047446d5ca
remove extra async execution
2022-01-17 15:50:11 -08:00
chrislu
7bf7af971b
more logs
2022-01-17 14:15:10 -08:00
chrislu
fc22071a2f
more logs
2022-01-17 14:02:37 -08:00
chrislu
381f4e73a0
delete actual reference first
2022-01-17 13:56:47 -08:00
chrislu
0ba88596e8
invalidate filehandle entry view cache
2022-01-17 13:53:30 -08:00
chrislu
1734017ba1
add test
2022-01-17 13:40:41 -08:00
chrislu
da7f13e73e
Revert "testing skip memory management"
...
This reverts commit 6c908352cb
2022-01-17 03:21:31 -08:00
chrislu
6c908352cb
testing skip memory management
2022-01-17 03:19:24 -08:00
chrislu
77d9993f38
remove unused variables
2022-01-17 03:19:11 -08:00
chrislu
f710d5ffca
a little speed up
2022-01-17 03:19:00 -08:00
chrislu
fc0628c038
working
2022-01-17 01:53:56 -08:00
chrislu
1bd6d289d4
better locking on file handle
2022-01-15 05:45:29 -08:00
chrislu
2bfeb5d1c8
add filer to iam option
2022-01-15 03:37:52 -08:00
chrislu
b17c426e99
weed server: optionally start IAM service
...
related to https://github.com/chrislusf/seaweedfs/issues/2560
2022-01-13 22:49:49 -08:00
chrislu
3c8b74318e
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-01-13 13:03:07 -08:00
chrislu
8907e6a40a
add more help messages
2022-01-13 13:03:04 -08:00
banjiaojuhao
45e9c83421
padding zero for sparse file
2022-01-13 22:21:22 +08:00
chrislu
fe5b9e39cc
POSIX: check permission when removing items
2022-01-13 02:07:39 -08:00
chrislu
1453263b63
remove dead code
2022-01-13 02:02:04 -08:00
chrislu
e69c374956
minor
2022-01-13 02:01:53 -08:00
chrislu
9b954dc0d4
adjust make file
2022-01-13 01:33:13 -08:00
chrislu
f2847f1266
POSIX: check deletion permission
2022-01-12 23:58:11 -08:00
chrislu
0c75f15062
POSIX: should not delete if a directory is not empty
2022-01-12 23:57:54 -08:00
chrislu
de27058d0b
POSIX: differentiate device and char device
2022-01-12 21:45:38 -08:00
chrislu
d400a11832
POSIX: adjust source file ctime
...
SeaweedFS uses mtime as ctime
2022-01-12 21:45:18 -08:00
chrislu
b44f05a2d0
POSIX: change timestamp on each attribute change
2022-01-12 19:31:25 -08:00
chrislu
15c01d8b7f
add some notes
2022-01-12 15:04:48 -08:00
chrislu
107a4884a8
shell: tighter memory allocation
2022-01-12 14:59:29 -08:00
chrislu
fec8428fd8
POSIX: different inode for same named different file types
2022-01-12 11:51:13 -08:00
chrislu
e82ad60122
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-01-12 11:07:49 -08:00
chrislu
caf0a3486b
POSIX: adjust ctime for file truncate
2022-01-12 11:07:39 -08:00
Konstantin Lebedev
edb753ab4d
https://github.com/chrislusf/seaweedfs/issues/2583
2022-01-12 16:04:59 +05:00
chrislu
adfd54e7c4
fix compilation
2022-01-12 01:24:24 -08:00
chrislu
6cc92817dc
add logs for request mode
2022-01-12 01:13:19 -08:00
chrislu
826a7b307e
master: remove hard coded filer settings in master.toml
...
fix https://github.com/chrislusf/seaweedfs/issues/2529
2022-01-12 01:11:25 -08:00
chrislu
cd1ad88f30
POSIX: check name is too long ENAMETOOLONG
2022-01-12 00:16:00 -08:00
chrislu
2dcb8cb93b
POSIX: ensure file and directory inodes are different
...
this is just an in memory representation.
POSIX wants different inode numbers for the same named file or directory.
2022-01-11 23:44:48 -08:00
chrislu
5bb37d5905
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-01-11 23:29:12 -08:00
chrislu
10ecf80ca1
add a debug capability to list all metadata keys
2022-01-11 23:25:04 -08:00
Kyle Sanderson
9e012001be
filer.copy: don't crash when volume creation fails
...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]
goroutine 7482 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x2)
/go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
/go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]
goroutine 7480 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x0)
/go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
/go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
2022-01-11 22:22:39 -08:00
chrislu
1a7d5b5b5e
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-01-11 12:24:56 -08:00
chrislu
41daecfdca
Update mount_std.go
2022-01-11 12:23:12 -08:00
chrislu
2d0ccc4d34
add logs
2022-01-11 12:23:01 -08:00
Chris Lu
abe5da7d2c
Merge pull request #2575 from Radtoo/fix_paths2
...
Fix paths2
2022-01-11 12:04:30 -08:00
chrislu
b8fbf19e9a
mount: rename follow POSIX
2022-01-11 03:23:03 -08:00
chrislu
6a12520a96
fix logging
2022-01-10 01:00:11 -08:00
chrislu
cbc055dc2b
mount: file fsync
...
fix https://github.com/chrislusf/seaweedfs/issues/2561
2022-01-10 00:52:16 -08:00
chrislu
19555385f7
2.85
2022-01-09 19:30:23 -08:00
Radtoo
389002f195
Using positional arguments rather than option flag to enable better shell usage
2022-01-08 16:52:12 +01:00
Radtoo
fba1efb77a
Now works with a single file too
...
Parsing removed from doFixOneVolume
Needle init removed from runFix
2022-01-08 16:31:53 +01:00
chrislu
110d5a5233
support fixing a collection of volumes, or volumes under one directory
2022-01-07 14:52:16 -08:00
chrislu
60dc450091
skip fixing read only volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/2562
2022-01-06 09:52:28 -08:00
chrislu
3df8f96117
avoid changing inode
2022-01-06 01:36:11 -08:00
chrislu
67b0645808
mount: need to change entry name after renaming
2022-01-05 21:27:41 -08:00
chrislu
4de060daa6
mount: skip special character in the filenames
...
fix https://github.com/chrislusf/seaweedfs/issues/2559
2022-01-05 03:57:24 -08:00
chrislu
e76105e2ab
fix auth permission checking
2022-01-03 21:05:20 -08:00
chrislu
a7887166cf
wildcard prefix to restrict access to directories in s3 bucket
...
https://github.com/chrislusf/seaweedfs/discussions/2551
2022-01-03 15:39:36 -08:00
chrislu
5799a20f71
2.84
2022-01-02 17:05:19 -08:00
Chris Lu
42c849e0df
Merge branch 'master' into metadata_follow_with_client_id
2022-01-02 01:07:30 -08:00
Chris Lu
9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158
...
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst
c35660175d
BUGFIX: ensure Authorization header is only added once
2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst
1cd3b6b4e1
BUGFIX: security.toml contained wrong keys
2021-12-31 22:05:41 +01:00
Sebastian Kurfuerst
10404c4275
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
...
- one JWT for reading and one for writing, analogous to how the JWT
between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer
Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.
## Docs to be adjusted after a release
Page `Amazon-S3-API`:
```
# Authentication with Filer
You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.
Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.
With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```
Page `Security Overview`:
```
The following items are not covered, yet:
- master server http REST services
Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.
...
Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**
...
# Securing Filer HTTP with JWT
To enable JWT-based access control for the Filer,
1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.
If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.
If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.
The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```
Page `Security Configuration`:
```
(update scaffold file)
...
[filer_jwt.signing]
key = "blahblahblahblah"
[filer_jwt.signing.read]
key = "blahblahblahblah"
```
Resolves : #158
2021-12-30 14:45:27 +01:00
chrislu
34742be029
remove duplicated metadata subscription in filer
...
https://github.com/chrislusf/seaweedfs/issues/2545
2021-12-30 01:51:52 -08:00
chrislu
5c87fcc6d2
add client id for all metadata listening clients
2021-12-30 00:23:57 -08:00
chrislu
fb434318e3
dynamically adjust connection timeout
...
better fix for https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:44:39 -08:00
chrislu
5788bf2270
s3: increase timeout limit
...
https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:21:02 -08:00
Sebastian Kurfuerst
fcc09cef6f
Refactor: pass in claim type into security.DecodeJwt
2021-12-29 12:40:41 +01:00
Sebastian Kurfuerst
d156d410ef
rename security.GenJwt to security.GenJwtForVolumeServer
2021-12-29 12:39:41 +01:00
Sebastian Kurfuerst
eda4c43a08
fix typo in error message
2021-12-29 12:38:14 +01:00
chrislu
498661e3bb
mount: remove limits on number of parallel requests
2021-12-28 17:41:01 -08:00
chrislu
9a00c17555
reader: avoid wrong pattern detection due to lock waiting
2021-12-28 16:30:33 -08:00
chrislu
f7a6f6b4c0
if this is enabled, there are some "bus error" with git clone
2021-12-28 13:44:52 -08:00
chrislu
0da2dfd640
fuse: change to direct io mode
...
before and after:
chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 4.534068 secs (236816430 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192 0.01s user 3.86s system 84% cpu 4.561 total
chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 3.824072 secs (280784948 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192 0.01s user 3.22s system 83% cpu 3.857 total
2021-12-28 12:22:56 -08:00
chrislu
3fd4da34a4
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2021-12-28 01:10:38 -08:00
chrislu
2422556456
monitor write pattern: avoid timing due to locking
2021-12-28 01:10:35 -08:00
chenkai
47c30e3add
filer list entries use context to break job
2021-12-28 15:03:41 +08:00
chrislu
80db8b13d8
bug: cleanup function was called twice
2021-12-27 20:53:02 -08:00
chrislu
67b723f74e
Filer Server API support fsync
...
fix https://github.com/chrislusf/seaweedfs/issues/2528
2021-12-26 17:28:47 -08:00
chrislu
9f9ef1340c
use streaming mode for long poll grpc calls
...
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
chrislu
c935b9669e
2.83
2021-12-25 01:01:34 -08:00
chrislu
eb4ad2546f
use proper chunk size limit option
2021-12-24 22:52:18 -08:00
chrislu
41bbf320bb
use 2MB chunk size. cache size is the wrong option
2021-12-24 22:50:19 -08:00
chrislu
982ea85d81
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2021-12-24 22:40:07 -08:00
chrislu
083d8e9ece
add stream writer
...
this should improve streaming write performance, which is common in many cases, e.g., copying large files.
This is additional to improved random read write operations: 3e69d19380...19084d8791
2021-12-24 22:38:22 -08:00
Sebastian Kurfürst
6db49100d6
BUGFIX: add access.ui setting to scaffolded security.toml
...
... The property is read here: b70cb3e0b2/weed/server/volume_server.go (L69)
2021-12-24 13:59:04 +01:00
chrislu
255a1c7dcd
refactor type names
2021-12-23 18:23:18 -08:00
chrislu
f77ca41769
refactor
2021-12-23 17:48:34 -08:00
chrislu
1d36884845
rename files
2021-12-23 17:47:58 -08:00
chrislu
2d1a1f5e03
rename variables and functions
2021-12-23 17:35:57 -08:00
chrislu
7bf48ee135
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2021-12-23 17:23:26 -08:00
chrislu
6de331b014
clean up
2021-12-23 17:23:21 -08:00
chrislu
032df784ed
chunked file works now
2021-12-23 17:17:32 -08:00
banjiaojuhao
083bf3a137
filer server: add "datacenter, rack and datanode" for path specific configuration
2021-12-23 23:25:05 +08:00
chrislu
c2aad1c7ff
detect non streaming mode on first write request
2021-12-22 17:20:44 -08:00
chrislu
b541e39a2c
fix tests
2021-12-22 16:17:30 -08:00
chrislu
2bc6fa90ff
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2021-12-22 16:05:43 -08:00
chrislu
0ec7bc6710
detect non streaming mode on the first read
2021-12-22 16:05:38 -08:00
chrislu
4c1368d621
fix test
2021-12-22 16:05:08 -08:00
Chris Lu
4e73705533
Merge pull request #2530 from banjiaojuhao/filer-upload-file-to-node
...
filer server: allow upload file to specific dataNode
2021-12-22 12:49:15 -08:00
banjiaojuhao
08336be92e
filer server: allow upload file to specific dataNode
2021-12-22 21:57:26 +08:00
chrislu
7b78fc72b0
add page chunk interval list
2021-12-22 02:53:33 -08:00
chrislu
b7cd52636b
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2021-12-21 22:24:43 -08:00
chrislu
3981d65b68
remove println
2021-12-21 22:24:38 -08:00
chrislu
9a73319b45
mount: different write strategy for streaming write and random write
2021-12-21 17:28:55 -08:00
Chris Lu
b0665a15f4
Merge pull request #2527 from banjiaojuhao/master-assign-by-datanode
2021-12-21 08:56:51 -08:00
banjiaojuhao
dda6b90d25
assign fileId according to DataNode with empty DataCenter and Rack
2021-12-21 17:28:33 +08:00
chrislu
4b8dcff448
reverting default admin scripts
...
fix https://github.com/chrislusf/seaweedfs/issues/2525
this new default value was introduced in 2.80
this affects production environments, e.g., EC is not desired, volume balancing is not preferred, etc.
2021-12-20 13:34:57 -08:00
chrislu
b21a67bbe6
add writer pattern object for later use
2021-12-20 11:53:48 -08:00
chrislu
4fd29dad86
remove writeOnly flag
2021-12-20 01:11:43 -08:00
chrislu
bc96682760
refactor, change file locations
2021-12-20 01:02:23 -08:00
chrislu
866c2657f0
avoid FUSE cache only for the first 512 bytes
2021-12-19 23:13:36 -08:00
chrislu
0cb9036f66
mount: only cache the first chunk on stream read
2021-12-19 23:06:03 -08:00
chrislu
a152f17937
mount: improve read performance on random reads
2021-12-19 22:43:14 -08:00
chrislu
85c526c583
s3: bind to a specific IP
...
fix https://github.com/chrislusf/seaweedfs/issues/2516
2021-12-17 11:34:37 -08:00
chrislu
5eacff9d4f
log message adds server name
...
address https://github.com/chrislusf/seaweedfs/issues/2514#issuecomment-995925733
2021-12-16 10:46:26 -08:00
chrislu
50ddd8c8e2
remove debug messages
...
fix https://github.com/chrislusf/seaweedfs/issues/2514
2021-12-16 00:58:15 -08:00
chrislu
7210558c7b
s3: pass through s3 presigned headers
...
fix https://github.com/chrislusf/seaweedfs/discussions/2502
2021-12-15 13:18:53 -08:00
chrislu
bf4d7affc0
gateway to remote object store: skip replicating multipart upload part files
...
fix https://github.com/chrislusf/seaweedfs/issues/2509
2021-12-14 19:48:31 -08:00
chrislu
316f326464
add more help message
2021-12-13 13:14:36 -08:00
Konstantin Lebedev
969f513265
disable audit log for s3 statusHandler
2021-12-13 16:08:19 +05:00
Chris Lu
ab0c6a84bf
Merge pull request #2503 from kmlebedev/audit_log_nonblocking
...
Audit log force async
2021-12-13 00:47:16 -08:00
Konstantin Lebedev
34779e8f38
force enable asynchronous I/O sending events to Fluentd
2021-12-13 13:39:39 +05:00
chrislu
c3b73ec23b
2.82
2021-12-12 23:25:24 -08:00
chrislu
7bf14452f7
s3: handle Response-Content-Disposition
2021-12-12 23:05:40 -08:00
chrislu
94e5c0a454
skip purging from ec volumes
2021-12-12 22:55:27 -08:00
chrislu
a2d3f89c7b
add lock messages
2021-12-10 13:24:38 -08:00
chrislu
991a3dca0d
rename file
2021-12-10 13:11:00 -08:00
kmlebedev
4f98553ba9
audit log SignatureVersion
2021-12-10 19:40:32 +05:00
chrislu
2d6fcdf83a
add delay if need to resume from disk
2021-12-09 13:13:05 -08:00
liubaojiang
c7abc34dbb
rename operation returns the correct old parent path
2021-12-10 00:02:57 +08:00
Konstantin Lebedev
98251fe16a
non blocking audit log
2021-12-09 19:47:16 +05:00
chrislu
40ee9b4d7b
sleep between the gap of each execution
...
related to https://github.com/chrislusf/seaweedfs/issues/2476
2021-12-07 11:44:08 -08:00
Chris Lu
2ba08afed1
Merge pull request #2498 from kmlebedev/s3_audit_log
2021-12-07 09:35:48 -08:00
Konstantin Lebedev
10678cde81
audit log config
2021-12-07 18:20:52 +05:00
Konstantin Lebedev
4ec8715f20
audit log
2021-12-07 12:15:48 +05:00
Konstantin Lebedev
3ac48cd540
audit log
2021-12-07 12:13:21 +05:00
chrislu
7f0a97c7b6
shell: volume.list supports different verbosity level
2021-12-05 21:54:40 -08:00
chrislu
5ea9715721
2.81
...
also sync java client version to SeaweedFS version
2021-12-05 18:05:24 -08:00
chrislu
e6c026db65
volume.fix.replication: fix misplaced volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/2416
2021-12-05 16:56:25 -08:00
chrislu
53e2dee177
[volume.check.disk] was using the wrong source and target locations
...
fix https://github.com/chrislusf/seaweedfs/issues/2268
2021-12-05 14:32:04 -08:00
chrislu
c146c76d10
avoid creating the same bucket with a different randomized name
...
related to https://github.com/chrislusf/seaweedfs/issues/2492
2021-12-05 13:06:41 -08:00
chrislu
42d97a3442
adjust randomized bucket name
2021-12-05 12:36:58 -08:00
chrislu
488afa5002
volume: load volume can optionally be skipped, if ec volume exists
...
fix https://github.com/chrislusf/seaweedfs/issues/2489
2021-12-05 02:28:52 -08:00
chrislu
59e58c4b23
volume: fix loading old volume format
...
fix https://github.com/chrislusf/seaweedfs/issues/2487
2021-12-05 01:06:01 -08:00
chrislu
b70cb3e0b2
upgrade protoc to 3.17.3
...
$brew install protobuf
$ protoc --version
libprotoc 3.17.3
$ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26
$ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1
2021-12-05 00:42:25 -08:00
chrislu
5c3b783310
s3: copy object API needs to escape special characters
...
fix https://github.com/chrislusf/seaweedfs/issues/2482
2021-12-04 23:24:53 -08:00
Bl1tz23
83bffca7e6
fix MongodbStore.ListDirectoryEntries panic on find failure
2021-12-03 13:58:37 +03:00
Tanmoy Majumdar
ea09fb477a
return ' shouldRetry=true' so that filer can retry the failed chunk
2021-12-03 11:54:20 +06:00
Chris Lu
689f5513a9
redis3 supports sentinel
2021-11-29 01:09:51 -08:00
Chris Lu
7227cfddf5
2.80
2021-11-29 00:57:08 -08:00
Chris Lu
3d7390302d
add s3.clean.uploads -timeAgo=24h
2021-11-29 00:49:49 -08:00
Chris Lu
2f72c24498
skip the rest logic
2021-11-29 00:32:21 -08:00
Chris Lu
88ff8fc27b
ensure uploaded chunks are deleted on error
2021-11-29 00:28:26 -08:00
Chris Lu
bea26549f2
avoid empty extended attribute value
2021-11-28 23:50:52 -08:00
Chris Lu
ce2af0082e
revert
2021-11-28 23:35:22 -08:00
Chris Lu
1c9f3c7ac0
read deleted chunks when replcating data
2021-11-28 23:34:34 -08:00
Chris Lu
96c66ca2aa
read deleted chunks when replicating data
2021-11-28 23:33:03 -08:00
Chris Lu
f3a334965d
Update command_remote_uncache.go
...
minor
2021-11-28 23:10:32 -08:00
Chris Lu
9ccfc1cfee
Update command_volume_fsck.go
...
minor
2021-11-28 22:39:24 -08:00
Chris Lu
71a94267c9
Update command_volume_fsck.go
...
remove unused code
2021-11-28 22:37:11 -08:00
Chris Lu
ad16221a35
adjust error log
2021-11-28 22:06:17 -08:00
Chris Lu
cf1586a34d
add logs for writing to remote file
2021-11-27 22:09:23 -08:00
Chris Lu
3a19eea97c
allocate memory by slabs
2021-11-27 12:13:00 -08:00
limd
8805c04128
fix redis2 sentinel config example
2021-11-25 19:20:02 +08:00
limd
ec03f22cc3
Merge remote-tracking branch 'origin/master'
2021-11-25 16:07:14 +08:00
limd
220797bd71
support redis sentinel
2021-11-25 15:57:03 +08:00
Chris Lu
6c27845be0
add retries when writing to remote s3
...
fix https://github.com/chrislusf/seaweedfs/issues/2465
2021-11-22 21:48:04 -08:00
Chris Lu
f3c789d662
2.79
2021-11-21 18:40:24 -08:00
Konstantin Lebedev
40abae7caa
Avoid forbiddening for HeadBucketHandler if owners are specified
...
https://github.com/chrislusf/seaweedfs/issues/2434
2021-11-17 14:42:24 +05:00
Chris Lu
3d87aa767d
fix same dc and other dc
2021-11-16 09:14:01 -08:00
Chris Lu
a0ef6e3611
prevent nil response
...
fix https://github.com/chrislusf/seaweedfs/issues/2452
2021-11-15 08:46:01 -08:00
Chris Lu
100c654ec3
2.78
2021-11-14 23:29:59 -08:00
Chris Lu
7bf891c00a
randomize same-dc servers and other-dc servers
2021-11-12 11:30:11 -08:00
Chris Lu
1f75f1f9dc
filer: fix mysql2 SQL template
2021-11-11 22:28:28 -08:00
Chris Lu
3abbaccb70
filer: fix mysql command to upsert
2021-11-11 22:27:13 -08:00
user
9668b15f38
Filtering by volume id is prioritized.
2021-11-11 18:18:56 +09:00
user
563a74a9eb
Volume filter by collection pattern added.
2021-11-11 18:01:47 +09:00
user
dbb8003ce3
Volume filter function added.
2021-11-11 17:45:17 +09:00
user
c387fe957b
Unused parameter removed.
2021-11-11 16:08:41 +09:00
Konstantin Lebedev
42f4aa824b
force overwrite s3-is-admin
...
https://github.com/chrislusf/seaweedfs/issues/2433
2021-11-11 11:34:01 +05:00
Chris Lu
c77c0de418
make install
2021-11-09 12:34:57 -08:00
Chris Lu
a7eee8b14b
Merge pull request #2427 from Bl1tz23/master
...
Fix: potencial iam identities data race
2021-11-09 08:53:48 -08:00
Chris Lu
1d4a61af5e
add brokers
2021-11-09 08:50:55 -08:00
Bl1tz23
c683409e92
s3: add RWMutex to iam, use RLock for concurrent reading
2021-11-09 18:11:06 +03:00
Bl1tz23
5018b22f36
s3: fix potencial iam identities data race
2021-11-09 12:19:50 +03:00
Chris Lu
59d1435d65
s3: avoid possible attacks by version
2021-11-08 17:47:56 -08:00
Chris Lu
5c99b8acc3
rename variables
2021-11-08 17:47:56 -08:00
Chris Lu
00fdab1284
adjust client name
2021-11-08 17:47:56 -08:00
Chris Lu
4729a57cc0
use constants
2021-11-08 17:47:56 -08:00
Chris Lu
d9dd72ea56
rename pacakge
2021-11-08 17:47:56 -08:00
Chris Lu
5cf332357b
2.77
2021-11-07 13:52:45 -08:00
Chris Lu
244841f080
adjust logs
2021-11-07 13:27:57 -08:00
Chris Lu
6e999f1176
s3: fix ListObject if more than 10000 objects
2021-11-07 12:39:36 -08:00
Chris Lu
a8b0f8864d
fix help message
2021-11-07 12:38:35 -08:00
Chris Lu
55973e8572
log unknown access key
2021-11-07 12:37:46 -08:00
Chris Lu
c4e22b5a9a
filer: deprecate "-peers" option
2021-11-06 14:36:45 -07:00
Chris Lu
751a7073e3
Merge branch 'flexible_filer_addition'
2021-11-06 14:26:26 -07:00
Chris Lu
e0fc2898e9
auto updated filer peer list
2021-11-06 14:23:35 -07:00
Chris Lu
04663c3611
remote.mount: print out metadata sync errors
2021-11-06 11:29:50 -07:00
Chris Lu
330d1fde7f
send peers info to filers
2021-11-06 11:29:50 -07:00
Chris Lu
84bb8e7365
send peers info to filers
2021-11-06 04:07:38 -07:00
Chris Lu
4b9c42996a
refactor grpc API
2021-11-05 18:11:40 -07:00
Chris Lu
5ea86ef1da
Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()"
...
This reverts commit af71ae11aa
2021-11-05 17:52:15 -07:00
Chris Lu
314c32514b
add command to list filers
2021-11-05 17:39:41 -07:00
Chris Lu
7348d2ae63
randomize traffic if replicated
2021-11-05 00:03:13 -07:00
Chris Lu
00ae965d8d
randomize a bit for ec shards distribution
2021-11-04 09:23:40 -07:00
Chris Lu
77f90ae288
add leader election in master
2021-11-04 00:54:38 -07:00
Chris Lu
35c37562bc
check auth only when enabled
2021-11-03 12:11:36 -07:00
Chris Lu
3c245c69d3
handle possible race condition
2021-11-03 01:36:39 -07:00
Chris Lu
af71ae11aa
master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()
2021-11-03 01:09:48 -07:00
Chris Lu
ab97b17e62
better printout
2021-11-02 23:45:47 -07:00
Chris Lu
e66865a8c5
adjust master client log level
2021-11-02 23:45:28 -07:00
Chris Lu
0c8dea9de8
go fmt
2021-11-02 23:39:16 -07:00
Chris Lu
5160eb08f7
shell: optionally read filer address from master
2021-11-02 23:38:45 -07:00
Chris Lu
18bfbf62fc
add one unit test
2021-11-02 23:36:43 -07:00
Chris Lu
e1ab8b01d0
s3: adjust permission for HEAD bucket operation
...
fix https://github.com/chrislusf/seaweedfs/issues/2417#issuecomment-958391856
2021-11-02 16:28:55 -07:00
Chris Lu
b25661c6df
s3: skip permission checking for creating bucket if the bucket already exists
...
fix https://github.com/chrislusf/seaweedfs/issues/2417
Rclone was trying to create the bucket even though the bucket already exists.
2021-11-02 13:13:36 -07:00
Chris Lu
1b90d60765
filer: paginate large directories to delete file chunks
2021-11-02 01:04:50 -07:00
Chris Lu
794375ca0a
adjust help message since both fullPercent and quietFor are needed.
2021-11-01 17:22:47 -07:00
Chris Lu
fc9e246592
2.76
2021-10-31 18:08:28 -07:00
Chris Lu
9cf756b188
S3: support CORS
...
related to https://github.com/chrislusf/seaweedfs/issues/1271
2021-10-31 18:06:43 -07:00
Chris Lu
f17fa400d5
refactoring
2021-10-31 18:05:34 -07:00
Chris Lu
006b4ec0f8
refactoring
2021-10-31 18:02:08 -07:00
Chris Lu
edbf6d297b
filer.meta.tail: add example to send metadata to elastic search
2021-10-31 18:01:33 -07:00
Chris Lu
24858507cc
rename API to avoid confusion
2021-10-30 19:27:25 -07:00
Chris Lu
f5824f369c
avoid reporting error when it is a user request error
2021-10-29 14:14:29 -07:00
Chris Lu
c857cc7286
cloud tier: remove tagging since not all s3 vendors support this
2021-10-29 12:39:19 -07:00
Chris Lu
d04cdcf40d
s3 header add user agent
2021-10-29 12:28:24 -07:00
Konstantin Lebedev
dc3ae5e8a6
DeleteBucketLifecycleConfiguration return 204
2021-10-28 19:34:37 +05:00
Konstantin Lebedev
2afb5a13af
fix DeleteBucketLifecycleConfiguration
2021-10-28 18:30:33 +05:00
Chris Lu
2e76834e4d
filer store: redis2 fix wrong pagination
2021-10-27 23:46:25 -07:00
Chris Lu
54b6e0f3fd
adjust logs
2021-10-27 23:46:07 -07:00
Chris Lu
900e888695
filer: avoid possible prefixed pagination loop on unsupported filer stores
2021-10-27 23:45:48 -07:00
Chris Lu
d774fa6c9a
rename variable
2021-10-25 14:39:20 -07:00
Chris Lu
2539ba0b62
fix compilation
2021-10-25 14:38:48 -07:00
Chris Lu
5f2d7c1589
erasure coding: skip erasure coding if less than recommended 4 nodes
2021-10-25 14:38:11 -07:00
Chris Lu
c9d3fb4a30
2.75
2021-10-24 18:15:59 -07:00
Chris Lu
5435027ff0
volume copy: stream out copying progress and avoid grpc request timeout
...
fix https://github.com/chrislusf/seaweedfs/issues/2386
2021-10-24 02:52:56 -07:00
Chris Lu
3be3c17f59
volume vacuum: avoid timeout with streaming progress report
...
fix https://github.com/chrislusf/seaweedfs/issues/2396
2021-10-24 01:55:34 -07:00
Konstantin Lebedev
235329a92a
fix Tagging add xmlns
2021-10-20 19:40:22 +05:00
Konstantin Lebedev
dc7e525cb9
fix Tagging test
2021-10-20 19:12:00 +05:00
Konstantin Lebedev
0b834600d5
fix PutObjectTaggingHandler Unmarshal
2021-10-20 17:58:06 +05:00
Konstantin Lebedev
c78220a7f2
fix object tagging https://github.com/chrislusf/seaweedfs/issues/2389
2021-10-20 16:01:06 +05:00
Chris Lu
182f43ae5f
2.74
2021-10-18 14:23:54 -07:00
Chris Lu
309f46e3af
S3: fix upload limit if the size is more than 8GB
2021-10-18 14:14:18 -07:00
Chris Lu
cd4fa7561b
2.73
2021-10-18 10:47:48 -07:00
Chris Lu
a6bb509cb7
adjust mime detection
2021-10-18 10:47:39 -07:00
Konstantin Lebedev
dd2cc1acaf
fix https://github.com/chrislusf/seaweedfs/issues/2387
2021-10-18 16:27:57 +05:00
Chris Lu
97c963bac9
2.72
2021-10-17 17:40:27 -07:00
Chris Lu
3833dac3f7
continue to read from memory if there is no flush
2021-10-17 13:53:04 -07:00
Chris Lu
29fa1b9fdc
add logs
2021-10-17 13:50:34 -07:00
Chris Lu
cca62fdb30
mount: streaming renaming folders
2021-10-17 04:22:42 -07:00
Chris Lu
004e56c1a6
save updated lastTsNs
2021-10-17 02:01:20 -07:00
Chris Lu
71a8c80f6f
s3: continuously listen to file updates
...
fix https://github.com/chrislusf/seaweedfs/issues/2342
2021-10-17 01:04:19 -07:00
Chris Lu
93bb7869b8
Revert "mount: fix renaming a deep directory with unvisited directories"
...
This reverts commit 0ccdb937bb
2021-10-16 23:52:30 -07:00
Chris Lu
8e2c9713a3
turn on new faster algorithm to translate into visible chunks
2021-10-16 23:35:20 -07:00
Chris Lu
e7d1f183cc
ensure file id is populated
2021-10-16 23:34:27 -07:00
Chris Lu
fc1694b321
add tests
2021-10-16 23:34:12 -07:00
Chris Lu
0ccdb937bb
mount: fix renaming a deep directory with unvisited directories
2021-10-16 23:33:45 -07:00
Chris Lu
b9a2efd69b
temporarily reverting
2021-10-16 16:29:10 -07:00
Chris Lu
8a124ef9fc
Revert "remove deprecated code"
...
This reverts commit de7688c539
2021-10-16 16:09:33 -07:00
Chris Lu
93624635b4
Revert "go fmt"
...
This reverts commit 6c704eb2ba
2021-10-16 16:09:30 -07:00
Chris Lu
6c704eb2ba
go fmt
2021-10-16 16:06:18 -07:00
Chris Lu
de7688c539
remove deprecated code
2021-10-16 16:06:12 -07:00
Chris Lu
7336990639
faster file read for large files
2021-10-16 16:03:16 -07:00
Chris Lu
8965a53c4d
add warning error
2021-10-16 15:57:30 -07:00
Chris Lu
8e3075d051
ensure file entry chunks have populated the file id
2021-10-16 01:05:48 -07:00
Chris Lu
822ee2475b
remove unnecessary file
2021-10-16 01:05:06 -07:00
Konstantin Lebedev
8de7915950
s3 mime detect
2021-10-14 15:03:11 +05:00
Chris Lu
5fd4b05c5e
Merge pull request #2381 from Juneezee/deprecate-ioutil
...
refactor: move from io/ioutil to io and os package
2021-10-13 22:38:58 -07:00
Chris Lu
46a09c6074
adjust test
2021-10-13 22:38:47 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package
...
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil . This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-14 12:27:58 +08:00
Chris Lu
4cbd390fbe
test: add fail message
2021-10-13 20:42:20 -07:00
Chris Lu
b693a8d6b9
Merge pull request #2380 from kmlebedev/bucket_policy
...
Bucket policy
2021-10-13 15:29:53 -07:00
Konstantin Lebedev
9d6ffa0ea1
GetBucketLifecycleConfigurationHandler
2021-10-14 01:35:33 +05:00
Konstantin Lebedev
458145425e
S3 MultipartUpload pass contentType to meta
2021-10-12 17:14:54 +05:00
Chris Lu
de1ec050e8
fier/s3: save "Content-Disposition" to extended properties
...
related to https://github.com/chrislusf/seaweedfs/issues/2371
2021-10-12 00:04:28 -07:00
Chris Lu
34f764007f
escape file name only when necessary
2021-10-11 23:33:45 -07:00
Chris Lu
8e9273db99
s3: use "response-content-disposition" to overwrite default content-disposition
...
fix one part of https://github.com/chrislusf/seaweedfs/issues/2371
see https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
2021-10-11 23:29:52 -07:00
Chris Lu
f0b928ff5e
go fmt
2021-10-11 23:23:46 -07:00
Chris Lu
3afa451cdc
volume: find a non-empty offset when binary searching by timestamp
2021-10-11 22:00:41 -07:00
Chris Lu
b530f12327
volume: find a non-empty offset when binary searching by timestamp
...
fix https://github.com/chrislusf/seaweedfs/issues/2364
2021-10-11 16:11:50 -07:00
Konstantin Lebedev
be4b3ed509
AclHandlers
2021-10-11 15:03:56 +05:00
Chris Lu
e6ef7b2387
return error early
...
fix https://github.com/chrislusf/seaweedfs/issues/2370
2021-10-11 01:24:30 -07:00
Chris Lu
3d586be552
2.71
2021-10-10 22:40:44 -07:00
Chris Lu
f4676824a7
fix test code compilation
2021-10-10 22:14:13 -07:00
Chris Lu
8205166009
disable testing with redis server
...
--- FAIL: TestNameList (0.00s)
panic: exec: "redis-server": executable file not found in $PATH [recovered]
panic: exec: "redis-server": executable file not found in $PATH
goroutine 37 [running]:
testing.tRunner.func1.2({0xde2f80, 0xc0003da160})
/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1209 +0x24e
testing.tRunner.func1()
/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1212 +0x218
panic({0xde2f80, 0xc0003da160})
/opt/hostedtoolcache/go/1.17.1/x64/src/runtime/panic.go:1038 +0x215
github.com/chrislusf/seaweedfs/weed/filer/redis3.TestNameList(0x407c59)
/home/runner/work/seaweedfs/seaweedfs/weed/filer/redis3/kv_directory_children_test.go:45 +0x376
testing.tRunner(0xc0003e2680, 0xf57f48)
/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1259 +0x102
created by testing.(*T).Run
/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1306 +0x35a
2021-10-10 21:57:55 -07:00
Chris Lu
3e2acf677c
removing tikv to resolve "go mod tidy" problem
...
tikv is causing "go mod tidy" problem. Need to resolve this before adding tikv back.
go mod tidy
go: finding module for package github.com/coreos/etcd/clientv3/balancer/picker
go: finding module for package cloud.google.com/go/kms/apiv1
go: finding module for package github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
go: finding module for package google.golang.org/grpc/naming
go: finding module for package github.com/coreos/etcd/clientv3/credentials
go: finding module for package github.com/coreos/etcd/clientv3/balancer
go: finding module for package github.com/d4l3k/messagediff
go: found github.com/coreos/etcd/clientv3/balancer in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/picker in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/resolver/endpoint in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/credentials in github.com/coreos/etcd v3.3.26+incompatible
go: found cloud.google.com/go/kms/apiv1 in cloud.google.com/go/kms v1.0.0
go: found github.com/d4l3k/messagediff in github.com/d4l3k/messagediff v1.2.1
go: finding module for package google.golang.org/grpc/naming
github.com/chrislusf/seaweedfs/weed/filer/tikv imports
github.com/tikv/client-go/v2/tikv imports
go.etcd.io/etcd/clientv3 tested by
go.etcd.io/etcd/clientv3.test imports
github.com/coreos/etcd/integration imports
github.com/coreos/etcd/proxy/grpcproxy imports
google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
2021-10-10 19:27:02 -07:00
Chris Lu
bf218cd59d
removing etcd sequencer
...
causing go mod tidy problem. If anyone wants this, please help to resolve this first.
github.com/chrislusf/seaweedfs/weed/sequence imports
go.etcd.io/etcd/client tested by
go.etcd.io/etcd/client.test imports
github.com/coreos/etcd/integration imports
github.com/coreos/etcd/proxy/grpcproxy imports
google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
2021-10-10 19:10:46 -07:00
Chris Lu
17e08a16f4
fix redis3 deletion
2021-10-09 04:54:48 -07:00
Chris Lu
ce1efeb0eb
adds more error message
2021-10-09 04:54:14 -07:00
Chris Lu
e4830bd93d
go fmt
2021-10-07 21:13:31 -07:00
Chris Lu
0a856241fe
avoid int bigger than math.MaxInt32
...
fix https://github.com/chrislusf/seaweedfs/issues/2363
2021-10-07 21:12:57 -07:00
Chris Lu
d688e10ed1
do not try to compress if not sure about the file content
2021-10-07 13:29:00 -07:00
Chris Lu
332f5ad3a8
revert temporary changes that disabled compression
...
fix https://github.com/chrislusf/seaweedfs/issues/2362
2021-10-07 13:24:16 -07:00
Chris Lu
f3d8232e14
reduce one redis lookup on hot path
2021-10-06 22:01:19 -07:00
Chris Lu
2336a397dc
use pipeline to save some time
2021-10-06 20:40:18 -07:00
Chris Lu
371fead8a5
redis3 using redis native sorted set
2021-10-06 18:18:24 -07:00
Chris Lu
8668d49c9d
test with real redis
...
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkNameList-12 1789 760599 ns/op
BenchmarkRedis-12 17539 64122 ns/op
PASS
2021-10-06 01:25:37 -07:00
Chris Lu
6b31f3c97a
add benchmark test
2021-10-06 00:37:57 -07:00
Chris Lu
893f0587b1
redis3 adds distributed locking
2021-10-06 00:03:54 -07:00
Chris Lu
f0d1e7bd05
skip ec volumes when loading normal volumes
2021-10-05 02:31:44 -07:00
Chris Lu
332d49432d
reduce concurrent volume grow requests
2021-10-05 01:58:30 -07:00
Chris Lu
96119eab00
refactor
2021-10-05 00:40:04 -07:00
Chris Lu
8a66306064
calculate disk usage in case of race condition
...
related to https://github.com/chrislusf/seaweedfs/issues/2357
2021-10-04 23:32:07 -07:00
Chris Lu
4ed2994555
use tsMemory to determine whether read from disk or memory
...
remove lastFlushTime
2021-10-04 16:02:56 -07:00
Chris Lu
6a030547a2
server: remove peer check if not starting master
...
more fix https://github.com/chrislusf/seaweedfs/issues/2352
2021-10-04 03:27:10 -07:00
Chris Lu
62c2732fd1
add back logic to check master peers
...
fix https://github.com/chrislusf/seaweedfs/issues/2352
2021-10-04 02:51:26 -07:00
Chris Lu
d4bb16e20e
Merge pull request #2354 from chrislusf/bptree
...
Add Redis3
2021-10-04 02:43:15 -07:00
Chris Lu
947add39e6
clean up *SkipListElementReference loaded from Redis
2021-10-04 02:31:38 -07:00
Chris Lu
513fed323a
SkipListElementReference can be an empty object
2021-10-04 02:30:44 -07:00
Chris Lu
280ab7f95c
add test
2021-10-04 02:30:24 -07:00
Chris Lu
04662126bb
add redis3
2021-10-04 01:04:27 -07:00
Chris Lu
2b9aab3442
use 1000 per batch
2021-10-04 01:03:40 -07:00
Chris Lu
366f522a2d
add redis3
2021-10-04 01:01:31 -07:00
Chris Lu
ba7fbac07f
rename
2021-10-03 19:23:34 -07:00
Chris Lu
e6196cdc50
add name list
2021-10-03 17:54:25 -07:00
Chris Lu
a481c4a45e
return previous element if visited
2021-10-03 13:50:52 -07:00
Chris Lu
22d8684e88
refactor out listStore
2021-10-03 02:19:21 -07:00
Chris Lu
d343b0db57
update value
2021-10-03 01:15:14 -07:00
Chris Lu
4f50f8c2ca
insert key and value
2021-10-03 01:07:35 -07:00
Chris Lu
69b84bb771
TestFindGreaterOrEqual
2021-10-02 14:15:49 -07:00
Chris Lu
57e2fd3f9b
remove bptree
2021-10-02 14:03:54 -07:00
Chris Lu
4c1741fdbb
working skiplist
2021-10-02 14:02:56 -07:00
Chris Lu
b297849147
typo
2021-10-01 23:24:54 -07:00
Chris Lu
af207bbaf0
retry both assign volume and uploading data
...
fix https://github.com/chrislusf/seaweedfs/issues/2351
2021-10-01 23:23:39 -07:00
Chris Lu
b6694279d7
Merge branch 'master' into bptree
2021-10-01 16:55:44 -07:00
Chris Lu
1e3fdf366f
go fmt
2021-10-01 12:10:24 -07:00
Chris Lu
e862b2529a
refactor
2021-10-01 12:10:11 -07:00
Chris Lu
88e006bc6d
Merge pull request #2349 from kmlebedev/fix_repl_volumes_per_step
...
Topology update for every Nth id volume
2021-10-01 12:05:51 -07:00
Konstantin Lebedev
5e64b22b45
check that the topology has been updated
2021-10-01 18:51:22 +05:00
Chris Lu
f58ea6a2ee
add source name to error message
2021-10-01 02:19:30 -07:00
chrislusf
3ffbaaa071
use github.com/linxGnu/grocksdb
2021-10-01 03:16:03 +00:00
Konstantin Lebedev
2cecde89c3
rename opt volumesPerStep
2021-10-01 00:17:54 +05:00
Konstantin Lebedev
fc51ffce2b
https://github.com/chrislusf/seaweedfs/issues/1846
2021-09-30 20:24:24 +05:00
Chris Lu
a067deaabc
avoid possible modified location list
...
fix issue 1 of https://github.com/chrislusf/seaweedfs/issues/2345
2021-09-28 16:54:18 -07:00
Chris Lu
040443e2d1
fix possible error case
2021-09-27 23:59:45 -07:00
Chris Lu
2e9372dcf7
volume stream read skips deleted content
2021-09-27 03:07:44 -07:00
Chris Lu
225b019fe0
stream read multiple volumes in a volume server
2021-09-27 02:51:31 -07:00
Chris Lu
1904448d4e
adjust starting offset
2021-09-27 02:01:29 -07:00
Chris Lu
5956a8b05a
adjust comment
2021-09-27 01:58:03 -07:00
Chris Lu
aa64f2ac4c
send needle data instead of raw needle body
2021-09-27 01:53:41 -07:00
Chris Lu
c4d7ee6c5c
volume server: read all files in a volume
2021-09-27 01:45:32 -07:00
Chris Lu
4a1d4d7462
s3: default to allow empty folder for better performance
2021-09-26 22:34:14 -07:00
Chris Lu
cee4d20bc1
2.70
2021-09-26 17:37:46 -07:00
Chris Lu
603ea2db73
avoid looping forever if there are no more metadata updates
2021-09-26 11:55:27 -07:00
Chris Lu
9887610b54
log tsNs should be processing time
2021-09-26 11:54:13 -07:00
Chris Lu
2baed2e1e9
avoid possible metadata subscription data loss
...
Previous implementation append filer logs into one file. So one file is not always sorted, which can lead to miss reading some entries, especially when different filers have different write throughput.
2021-09-25 01:18:44 -07:00
Chris Lu
a814f3f0a8
adjust metadata tail output
2021-09-25 01:04:51 -07:00
Chris Lu
52fe86df45
use default 10000 for grpc port
2021-09-20 14:05:59 -07:00
Chris Lu
b3d88180ca
Merge branch 'master' into bptree
2021-09-19 23:56:59 -07:00
Chris Lu
fa7c65bd4b
2.69
2021-09-19 21:44:06 -07:00
Chris Lu
84fdda85e7
go fmt
2021-09-19 12:06:15 -07:00
Chris Lu
ad5099e570
refactor
2021-09-19 12:02:23 -07:00
Chris Lu
5abdc0be77
s3: avoid overwriting object with ACL/LegalHold/Retension/LockConfiguration requests
2021-09-19 03:24:47 -07:00
Chris Lu
ede7a65a50
update log
2021-09-19 01:33:50 -07:00
Chris Lu
c1255d30ad
add cache error log
2021-09-19 01:33:26 -07:00
Chris Lu
59dd271734
more glog
2021-09-19 00:29:51 -07:00
Chris Lu
4cbba2b1c6
add more glog to s3
2021-09-19 00:28:22 -07:00
Chris Lu
71175461ef
add glog for s3 handlers
2021-09-19 00:18:59 -07:00
Chris Lu
e066e2642c
add NodeStore
2021-09-18 15:32:17 -07:00
Chris Lu
198fa58e3c
Merge branch 'master' into bptree
2021-09-18 14:06:16 -07:00
Chris Lu
49d971e602
filer: redis store needs to clean up batch deleted sub folders
2021-09-18 14:05:16 -07:00
Chris Lu
8f2e4be074
wip
2021-09-18 14:04:30 -07:00
Chris Lu
b751debd31
split node based on the last inserted key
2021-09-18 01:29:47 -07:00
Chris Lu
2226c3c8b6
Merge branch 'master' into bptree
2021-09-17 10:35:21 -07:00
Chris Lu
94a01fcfcb
filer.remote.gateway: add options to include or exclude new bucket names to mirror
2021-09-15 23:04:16 -07:00
Chris Lu
b5f4910412
adjust help messages
2021-09-15 22:53:10 -07:00
Chris Lu
f0907eb83c
add command filre.remote.gateway
2021-09-15 22:48:04 -07:00
Chris Lu
63da4bbb54
separate filer.remote.gateway command to avoid confusion
2021-09-15 22:47:17 -07:00
Chris Lu
2789d10342
go fmt
2021-09-14 10:37:06 -07:00
Chris Lu
119d5908dd
shell: do not need to lock to see volume -h
2021-09-13 22:13:34 -07:00
Chris Lu
7504be58f9
Avoid xattr printed out as HTTP headers
...
fix https://github.com/chrislusf/seaweedfs/issues/2336
2021-09-13 04:00:57 -07:00
Chris Lu
96514f0f00
fix tests
2021-09-13 02:19:48 -07:00
Chris Lu
20ac710ceb
2.68
2021-09-13 02:16:09 -07:00
Chris Lu
6cd1ce8b74
erasure coding: add cleanup step if anything goes wrong
2021-09-13 01:55:49 -07:00
Chris Lu
f74b29416a
better etag matching
2021-09-13 00:31:46 -07:00
Chris Lu
e9760f261e
minor
2021-09-13 00:31:06 -07:00
Chris Lu
e5fc35ed0c
change server address from string to a type
2021-09-12 22:47:52 -07:00
Chris Lu
2c9d4c8f43
custom grpc port: filer
2021-09-12 02:28:37 -07:00
Chris Lu
e690a2be16
custom grpc port: volume server
2021-09-12 02:25:15 -07:00
Chris Lu
232ad2fe65
custom grpc port: master follower
2021-09-12 02:25:00 -07:00
Chris Lu
bd1f800efe
custom grpc port: master
2021-09-12 02:19:10 -07:00
Chris Lu
7591336a22
log format
2021-09-11 14:27:57 -07:00
Chris Lu
8c6ff55226
add volume not found error type, to reduce error log
2021-09-11 14:26:41 -07:00
Chris Lu
7e600bff5f
adjust log level
2021-09-11 12:59:15 -07:00
Chris Lu
a243d7e047
EC: clean up if failed in the middle
2021-09-11 02:07:14 -07:00
Chris Lu
f7f01ac209
return if error
2021-09-11 02:06:43 -07:00
Chris Lu
59cb6146a3
refactor
2021-09-11 02:06:30 -07:00
Chris Lu
5496d68f6a
increase counter only if not early terminated
2021-09-11 02:05:55 -07:00
Chris Lu
2b8ab1534a
mysql table DDL: minor, not necessary, but a little more strict
...
the directory is already hashed into dirhash. So this change is not strictly required.
2021-09-08 23:55:18 -07:00
joshuafc
5654d0d60d
CompactionTableSizeMultiplier of leveldb use default value. #2325
...
To improve performance of leveldb find key in condition of large directory(millions of files) which use uuid as filename.
2021-09-09 10:42:34 +08:00
Chris Lu
0b4269b6a8
remember commands even if failed
2021-09-08 15:55:19 -07:00
Chris Lu
0207f5fe9b
replicated remote.cache
2021-09-08 15:54:55 -07:00
Chris Lu
00edcf71fc
fix compilation
2021-09-08 07:27:03 -07:00
Chris Lu
574485ec69
better IP v6 support
2021-09-07 19:29:42 -07:00
Chris Lu
0128239c0f
handle ipv6 addresses
2021-09-07 16:43:54 -07:00
zhoub
6a7ed1bd0e
add bloom filter to leveldb_store to improve fuse performance.
2021-09-07 21:09:10 +08:00
Chris Lu
9fdf02bcda
remove detecting ipv6
...
Got this error on my local:
transport: Error while dialing dial tcp [fe80::1]:19333: connect: no route to host
related to https://github.com/chrislusf/seaweedfs/pull/2310
2021-09-07 02:31:34 -07:00
Chris Lu
889a39a24f
ipv6
2021-09-07 02:30:25 -07:00
Chris Lu
8c6d706328
2.67
2021-09-07 00:08:03 -07:00
Chris Lu
2b1feb732c
remote.cache supports replication
2021-09-06 18:30:44 -07:00
Chris Lu
3adc3da291
refactor
2021-09-06 16:35:55 -07:00
Chris Lu
6923af7280
refactoring
2021-09-06 16:20:49 -07:00
Chris Lu
64f6532fbe
update help message
2021-09-06 16:16:22 -07:00
Chris Lu
308d4b2dae
filer.remote.sync: createBucketWithRandomSuffix defaults to true
2021-09-06 15:14:26 -07:00
Chris Lu
8e4b43a017
minor
2021-09-06 15:13:38 -07:00
Chris Lu
c218ef20c7
filer.remote.sync: automatically detect the primary remote storage
2021-09-06 15:10:55 -07:00
Chris Lu
1702ce5395
remove unused code
2021-09-06 12:01:44 -07:00
Chris Lu
0cda61d539
return error if not found
2021-09-05 23:43:44 -07:00
Chris Lu
f62df7f695
fix wrong srcPath checking
2021-09-05 23:37:40 -07:00
Chris Lu
e93d4935e3
add other replica locations when assigning volumes
2021-09-05 23:32:25 -07:00
Chris Lu
7a13816e94
refactor
2021-09-05 23:17:15 -07:00
Chris Lu
6022db6d6a
2.66
2021-09-05 16:21:14 -07:00
Chris Lu
006c01a519
fix format
2021-09-05 16:18:50 -07:00
Chris Lu
65af3cf4df
master: disconnect only the phantom volume server
...
fix https://github.com/chrislusf/seaweedfs/issues/2311
2021-09-05 15:20:03 -07:00
Chris Lu
57a95887d2
remote.cache remote.uncache supports all mounted directories
2021-09-05 14:47:06 -07:00
Chris Lu
28f45f8fa6
re-org
2021-09-05 14:38:31 -07:00
Chris Lu
c735608685
obfuscate secret keys on display
2021-09-05 14:23:49 -07:00
Chris Lu
32e94de86a
default auto trimming suffix to true
2021-09-05 13:27:35 -07:00
Chris Lu
60573fd3e2
option to map remote bucket to trimmed bucket name
2021-09-05 11:55:52 -07:00
Chris Lu
2348e8d8da
Merge pull request #2310 from nivekuil/ipv6
...
Detect ipv6 addresses
2021-09-05 10:56:44 -07:00
nivekuil
0fe9d2997b
Detect ipv6 addresses
2021-09-05 06:21:40 -07:00
Chris Lu
bdefdee4e6
filer.remote.sync: add option to add randomized suffix to buckets to avoid conflicts
2021-09-04 22:46:28 -07:00
Chris Lu
d57d4c5f8f
shell: add remote.mount.buckets
2021-09-04 21:37:25 -07:00
Chris Lu
98c68d7a79
filer.remote.sync fix upload logic
2021-09-04 18:46:28 -07:00
Chris Lu
d983aa4c7d
correct filtering
2021-09-04 13:58:14 -07:00
Chris Lu
c5ee03d6af
format
2021-09-04 13:57:55 -07:00
Chris Lu
cc278a5928
Update Makefile
2021-09-04 13:46:49 -07:00
Chris Lu
16a342eb8a
re-ordering
2021-09-04 13:46:44 -07:00
Chris Lu
796b2eb929
remember sync time
2021-09-04 13:46:22 -07:00
Chris Lu
53b9b521c9
adjust error message
2021-09-04 13:46:06 -07:00
Chris Lu
49952ba905
fix cache/uncache filters
2021-09-04 13:45:39 -07:00
Chris Lu
99d4594ee8
update remote storage mount mapping
2021-09-04 05:22:34 -07:00
Chris Lu
df29281536
rename
2021-09-04 05:11:06 -07:00
Chris Lu
c3db389e42
refactor
2021-09-04 05:09:38 -07:00
Chris Lu
e9ce099453
rename
2021-09-04 05:03:51 -07:00
Chris Lu
8707ef00a6
refactor
2021-09-04 05:02:20 -07:00
Chris Lu
13cc2ef723
when creating bucket, use remote storage client for default storage
2021-09-04 04:54:55 -07:00
Chris Lu
63c84584bb
refactor
2021-09-04 04:53:36 -07:00
Chris Lu
4b28c5f6c3
filer.remote.sync: split into buckets mode and single directory mode
2021-09-04 04:35:46 -07:00
Chris Lu
da49d25950
auto list of storage types
2021-09-04 00:18:21 -07:00
Chris Lu
38f73c8324
add gcs project id
2021-09-03 23:38:33 -07:00
Chris Lu
8ec357b3d3
go mod
2021-09-03 23:25:33 -07:00
Chris Lu
bbc77f7af4
fix compilation
2021-09-03 22:56:59 -07:00
Chris Lu
0652805236
cloud drive: add createBucket() deleteBucket()
2021-09-03 22:30:55 -07:00
Chris Lu
03a31587ce
go fmt
2021-09-03 20:42:28 -07:00
Chris Lu
83cd0fc739
cloud drive: add list buckets
2021-09-03 20:42:02 -07:00
Chris Lu
fbfc90fd1e
adjust formatting remote location
2021-09-03 18:52:37 -07:00
Chris Lu
491f7636f8
s3: CopyObject return http Status 400 Bad Request for non-existing source
...
fix https://github.com/chrislusf/seaweedfs/issues/2306
2021-09-03 17:38:39 -07:00
Chris Lu
bca4a9de78
simplify
2021-09-02 23:09:24 -07:00
Chris Lu
958125bd02
conforming to http user agent common practice
2021-09-02 22:55:35 -07:00
Chris Lu
11a496404b
reset wait time
2021-09-02 19:55:01 -07:00
Chris Lu
7f2a8246a9
minor
2021-09-02 12:32:59 -07:00
Chris Lu
1b942dfa3c
add error message
2021-09-02 12:32:46 -07:00
Chris Lu
9a73b0e3c9
refactor
2021-09-02 07:07:16 -07:00
Chris Lu
e281f0fa82
refactor
2021-09-02 06:53:21 -07:00
Chris Lu
24fda6f47f
remove QR code for mobile upload
2021-09-01 16:59:37 -07:00
Chris Lu
7ce97b59d8
go fmt
2021-09-01 02:45:42 -07:00
Chris Lu
d1a4e19a3f
volume: copy file also copies modification time
...
to ensure ttl can work well
2021-09-01 02:42:57 -07:00
Chris Lu
3bd48c4f29
filer.remote.sync: exit when directory is unmounted
...
this will not propagate the deletions back to the cloud
2021-09-01 01:29:22 -07:00
Chris Lu
3faaa6e360
ensure cached client with updated storage conf
2021-09-01 01:27:45 -07:00
Chris Lu
8e125339d5
line wrap
2021-08-31 23:30:28 -07:00
Chris Lu
68ea99190f
add additionalPathPrefixes
2021-08-31 23:26:26 -07:00
Chris Lu
43fd11278e
support follow additional path prefixes
2021-08-31 23:23:08 -07:00
Chris Lu
cca588429b
shell: fs.meta.save skip saving system logs
2021-08-31 23:05:56 -07:00
Chris Lu
0ee208c1a4
adjust example
2021-08-31 02:04:32 -07:00
Chris Lu
97b7948cd9
adjust error message
2021-08-31 01:07:34 -07:00
Chris Lu
c9205458a1
cloud drive: create mount directory if not exists
2021-08-31 01:07:19 -07:00
Chris Lu
1dfcdc53b7
cloud drive: add storj, filebase
2021-08-30 17:43:14 -07:00
Chris Lu
701cc133f3
cloud drive: s3 add option for v4 signature
2021-08-30 17:28:33 -07:00
Chris Lu
f117f7bf54
adjust comment
2021-08-30 15:56:14 -07:00
Chris Lu
72d4ff6b28
Merge pull request #2280 from blacktear23/filerstore-tikv
...
Use TiKV as Filerstore
2021-08-30 02:21:14 -07:00
Chris Lu
f811fd0903
fix build
2021-08-30 02:03:08 -07:00
yulai.li
b17b81529e
Add build tags for TiKV filerstore
2021-08-30 15:59:25 +08:00
Rain Li
e5f9ff983d
Merge branch 'master' into filerstore-tikv
2021-08-30 15:52:54 +08:00
Chris Lu
44d0ac95f6
add cluster id if already set
2021-08-29 22:19:46 -07:00
Chris Lu
18bf556d93
fix comments
2021-08-29 22:19:25 -07:00
Chris Lu
21f37a0a75
remove unused code
2021-08-29 22:19:11 -07:00
Chris Lu
5adfdd7982
change to a new bloom filter library
2021-08-29 21:26:38 -07:00
Chris Lu
05f32376eb
add cluster id in filer configuration response
2021-08-29 21:02:10 -07:00
Chris Lu
eacaa44dc2
refactor
2021-08-29 19:13:48 -07:00
Chris Lu
6deee4c0b9
import hdfs
2021-08-29 18:49:43 -07:00
Chris Lu
a31f2907f0
cloud drive: filer.remote.sync supports remove folder
2021-08-29 18:46:28 -07:00
Chris Lu
001a472057
cloud mount: remote storage support hdfs
2021-08-29 18:41:29 -07:00
Chris Lu
05d2774117
refactoring
2021-08-28 22:49:38 -07:00
Chris Lu
975fbc9ee3
add wasabi example
2021-08-28 22:49:38 -07:00
Chris Lu
bec3f63298
2.65
2021-08-28 05:27:33 -07:00
Chris Lu
223c1b3571
adjust invalidation logic
2021-08-28 05:21:01 -07:00
nivekuil
8c523d50ad
Invalidate mount meta cache more aggressively
2021-08-28 04:26:37 -07:00
Chris Lu
65a29ad42a
add warning on unmount a folder
2021-08-28 02:25:48 -07:00
Chris Lu
a22f37b01c
remove tencent region
2021-08-28 02:23:03 -07:00
Chris Lu
a9b41900d8
handle cases when mounted directory does not exist
2021-08-28 01:56:56 -07:00
Chris Lu
0d8936f6cd
add aliyun example
2021-08-28 01:09:48 -07:00
Chris Lu
dc481c081c
remove unused function
2021-08-27 22:18:54 -07:00
Chris Lu
9242f3aaeb
fix build
2021-08-26 17:33:57 -07:00
Chris Lu
49a8dfb976
adjust default concurrent level
2021-08-26 17:05:56 -07:00
Chris Lu
6a0bb7106b
cloud drive: parallelize remote storage downloading
2021-08-26 16:16:26 -07:00
Chris Lu
05a648bb96
refactor: separating out remote.proto
2021-08-26 15:18:34 -07:00
yulai.li
c1dc5ab4ac
Add deleterange_concurrency to filer configuration file
2021-08-26 18:25:08 +08:00
yulai.li
318757ef8c
Change DeleteFolderChildren to DeleteRange api
2021-08-26 17:49:56 +08:00
yulai.li
de8ef28460
Update filer.toml scaffold to support tikv store
2021-08-26 16:26:27 +08:00
yulai.li
546efeba8f
Fix build bug
2021-08-26 16:20:35 +08:00
yulai.li
2088f28424
init post
2021-08-26 15:20:18 +08:00
Chris Lu
c08ac536ed
cloud drive: add support for Wasabi
...
* disable md5, sha256 checking to avoid reading one chunk twice
* single threaded upload to avoid chunk swapping (to be enhanced later)
2021-08-25 17:34:29 -07:00
Chris Lu
9bcf94b2b1
ensure multi-threaded correctness
2021-08-25 17:28:50 -07:00
Chris Lu
a7a914f120
cloud drive: add support for Baidu BOS
2021-08-24 23:46:33 -07:00
Chris Lu
2d805beed4
add help mesages
2021-08-24 23:28:35 -07:00
Chris Lu
c6c97bd83d
add default env variables
2021-08-24 23:25:36 -07:00
Chris Lu
a19c728034
cloud drive: add support for Tencent COS
2021-08-24 23:19:45 -07:00
Chris Lu
19a81d25af
cloud drive: add support for Aliyun OSS
2021-08-24 23:14:24 -07:00
Chris Lu
47d775cf68
cloud drive: add support for BackBlaze
2021-08-24 22:30:06 -07:00
Chris Lu
6bab20d862
cat entry content
2021-08-24 02:38:32 -07:00
Chris Lu
e9ebe24f2e
cloud drive: add support for Azure
2021-08-24 01:18:30 -07:00
Chris Lu
7c39a18ba5
update azure library
2021-08-24 00:32:35 -07:00
Chris Lu
a78d0227cd
adjust package name
2021-08-23 23:19:31 -07:00
Chris Lu
12631a3f5b
cloud drive: gcs simplify a little bit
2021-08-23 14:43:01 -07:00
Chris Lu
00c4e06caa
cloud drive: s3 configurable force path style
2021-08-23 03:30:41 -07:00
Chris Lu
f0cc130849
do not force path style for better compatibility
2021-08-23 03:09:41 -07:00
Chris Lu
2836a58d87
cloud drive: S3 supports storage class
2021-08-23 02:18:59 -07:00
Chris Lu
95e2b83ca5
fix format
2021-08-23 00:49:59 -07:00
Chris Lu
ff7dc3b44c
2.64
2021-08-23 00:39:15 -07:00
Chris Lu
ca35a77e85
adjust help message
2021-08-23 00:37:55 -07:00
Chris Lu
258063de26
cloud drive: add google cloud storage
2021-08-23 00:29:27 -07:00
Chris Lu
df1d6133a8
bptree does not work well for auto-increasing keys
2021-08-22 18:19:26 -07:00
Chris Lu
3b2c39f1be
mount: set name when mount path equals mount folder
...
fix https://github.com/chrislusf/seaweedfs/issues/2275#issuecomment-903255876
2021-08-22 04:59:09 -07:00
Chris Lu
51c8f2518f
change key type to ItemKey
2021-08-21 15:54:42 -07:00
Chris Lu
b3e49d2758
change value type to ItemValue
2021-08-21 15:52:17 -07:00
Chris Lu
38c8470d1d
add back non_dedup
2021-08-21 15:13:13 -07:00
Chris Lu
849f185a20
add memory kv store
2021-08-21 15:00:44 -07:00
Chris Lu
5f6cc9a814
make proto node
2021-08-21 13:36:52 -07:00
Chris Lu
2158d4fe4d
adjust help message
2021-08-21 02:17:10 -07:00
Chris Lu
172da83449
bpnode use get prev and next
2021-08-20 18:50:16 -07:00
Chris Lu
01661ec6a7
move to getter setter file
2021-08-20 18:37:34 -07:00
Chris Lu
0c360eb6b2
add getter and setter for root of tree and map
2021-08-20 18:34:50 -07:00
Chris Lu
1b1ab331f6
Merge pull request #2274 from qieqieplus/gzip-pool
2021-08-20 04:22:41 -07:00
Chris Lu
88d68cad87
remove dedup
2021-08-20 04:14:52 -07:00
qieqieplus
7720533f84
reduce gzip allocation
2021-08-20 18:38:18 +08:00
Chris Lu
2d237da74a
remove size since each put/get will have to update the root node
2021-08-20 01:19:11 -07:00
Chris Lu
ec72547c8d
started by copying from https://sourcegraph.com/github.com/timtadh/data-structures@master/-/tree/tree/bptree
2021-08-20 01:12:52 -07:00
Chris Lu
05fc7db755
filer.copy: large files pack file ids into chunks
2021-08-18 23:46:54 -07:00
mauroparente
c63713774b
Fix image resize
...
https://github.com/chrislusf/seaweedfs/issues/2262
2021-08-18 23:15:35 +02:00
Chris Lu
570d2eccec
skip body if not allowed by http status
2021-08-18 00:56:35 -07:00
Chris Lu
ccb3df41f8
elide secret key
2021-08-17 11:27:08 -07:00
Konstantin Lebedev
865b06b7fe
The IEEE assembler optimizations has been submitted and will be part of the Go 1.6 standard library.
...
https://github.com/klauspost/crc32
2021-08-17 13:06:48 +05:00
Chris Lu
e2aa3cf63b
fix go test
2021-08-15 23:20:46 -07:00
Chris Lu
1f35d32be0
2.63
2021-08-15 23:14:59 -07:00
Chris Lu
40dc283b2d
fix locating data chunks
2021-08-15 23:07:58 -07:00
Chris Lu
72eb6d5b9d
ensure no writes to remote storage if content is not changed
2021-08-15 20:23:41 -07:00
Chris Lu
5a7c40510f
format output
2021-08-15 20:07:13 -07:00
Chris Lu
bb94930196
add some delays if error
2021-08-15 20:06:47 -07:00
Chris Lu
5d5a21ba2d
adjust log format
2021-08-15 19:46:45 -07:00
Chris Lu
c45c12aedc
metadata updates
2021-08-15 19:27:30 -07:00
Chris Lu
507411bdf0
fix https://github.com/chrislusf/seaweedfs/issues/2260
...
fix https://github.com/chrislusf/seaweedfs/issues/2260
2021-08-15 19:16:00 -07:00
Chris Lu
c3ffd457ef
fix compilation error
2021-08-15 12:40:22 -07:00
Chris Lu
49b5e47bd1
retry forever with filer.remote.sync, and some refactoring
2021-08-15 12:38:26 -07:00
Chris Lu
fda2fc47b1
add RetryForever
2021-08-15 12:37:35 -07:00
Chris Lu
a539d64896
refactor
2021-08-15 12:09:54 -07:00
Chris Lu
8f7d2d317f
readerAt need to use the right offset
...
fix https://github.com/chrislusf/seaweedfs/issues/2259
2021-08-15 11:55:58 -07:00
Chris Lu
0b2a92d371
skip if http.StatusNoContent
2021-08-15 11:27:49 -07:00
Chris Lu
ec989b0377
remove println
2021-08-15 02:10:27 -07:00
Chris Lu
9462f5129a
shell: add "remote.meta.sync"
2021-08-15 01:53:46 -07:00
Chris Lu
3ada61c875
rename
2021-08-14 21:50:35 -07:00
Chris Lu
c34747c79d
rename, fix wrong logic.
2021-08-14 21:46:34 -07:00
Chris Lu
cb53802752
adjust help message
2021-08-14 15:55:53 -07:00
Chris Lu
889b143fa7
adjust modification detection logic
2021-08-14 15:44:47 -07:00
Chris Lu
f365af81c2
parallelize remote content fetching
2021-08-14 15:41:37 -07:00
Chris Lu
53e66980b2
add comments
2021-08-14 15:16:10 -07:00
Chris Lu
9921801e0c
Revert "use default or path-specific setting for cache replication level"
...
This reverts commit ba6923b223
2021-08-14 15:14:26 -07:00
Chris Lu
ba6923b223
use default or path-specific setting for cache replication level
2021-08-14 15:14:01 -07:00
Chris Lu
708debca14
remote.cache and uncache: more flexible options to select files to cache or uncache
2021-08-14 15:11:55 -07:00
Chris Lu
08258a819d
fix mistake
2021-08-14 05:10:30 -07:00
Chris Lu
4909bd9684
gRpc connection error on filer when no volume left #2243
...
fix https://github.com/chrislusf/seaweedfs/issues/2243
grpc do not cache connections only when connection problem happens.
Normal error results should not close the shared grpc connection.
2021-08-14 05:06:44 -07:00
Chris Lu
8126ab4b5d
rename
2021-08-14 05:03:45 -07:00
Chris Lu
0db2517994
go fmt
2021-08-14 02:55:44 -07:00
Chris Lu
7937db52e1
Filer locationPrefix configure does not exec replication #2257
...
fix https://github.com/chrislusf/seaweedfs/issues/2257
2021-08-14 02:54:13 -07:00
Chris Lu
0c66b173a4
fix
2021-08-13 11:31:43 -07:00
Chris Lu
e02a317d3d
adjust retry logic in case some data is partially written
2021-08-13 11:30:38 -07:00
Chris Lu
2d519c6cb6
adjust the retry logic
2021-08-13 11:13:30 -07:00
Chris Lu
b961fcd338
filer: stream read from volume server, reduce memory usage
2021-08-13 11:00:11 -07:00
Chris Lu
f4decf02df
volume copying: clean up stale volume data files
...
fix https://github.com/chrislusf/seaweedfs/issues/2250
2021-08-13 03:24:21 -07:00
Chris Lu
0f7d4556d8
shell: volume.tier.move makes up changes if volume move failed
2021-08-13 03:09:28 -07:00
Chris Lu
333cdce485
add verbose message
2021-08-13 02:57:14 -07:00
Chris Lu
78e8ddf910
Only when tailing volume, the zero-ed cookie should skip checking.
...
This only happens when checkCookie == false and fsync == false.
2021-08-13 02:09:35 -07:00
Chris Lu
a8617c1a39
tail volume: fix zero cookie problem from batch deletion
2021-08-13 01:54:35 -07:00
Chris Lu
5a0f92423e
use grpc and jwt
2021-08-12 21:40:33 -07:00
Chris Lu
6238644c35
remove gateway command
2021-08-12 20:52:04 -07:00
Chris Lu
d1d1fc772c
move some volume lookup operations to grpc
...
jwt related lookup will come in next commit
2021-08-12 20:33:00 -07:00
Chris Lu
5571f4f70a
master: add master.follower to handle read file id lookup requests
2021-08-12 18:10:59 -07:00
Chris Lu
5469019852
adjust data type
2021-08-12 17:54:34 -07:00
Chris Lu
bfac55e6c0
avoid integer overflow
...
fix https://github.com/chrislusf/seaweedfs/issues/2254
2021-08-11 22:22:49 -07:00
Chris Lu
5516fa8e80
Merge pull request #2251 from kmlebedev/rertyUpdateIamIdentity
...
Retry save and update IAM identity
2021-08-11 09:21:03 -07:00
Konstantin Lebedev
f0afd35eec
Retry save and update IAM identity
...
https://github.com/chrislusf/seaweedfs/issues/2242
2021-08-11 19:29:04 +05:00
byunghwa.yun
775dfbae85
Synchronize number of open files
2021-08-11 23:14:56 +09:00
Konstantin Lebedev
ec09966fd3
Retry save and update IAM identity
...
https://github.com/chrislusf/seaweedfs/issues/2242
2021-08-11 17:53:48 +05:00
Chris Lu
f2cd753bf9
fix avoid lock error
...
fix https://github.com/chrislusf/seaweedfs/issues/2247
2021-08-10 14:34:13 -07:00
Chris Lu
01336d71eb
minor
2021-08-10 13:04:33 -07:00
Chris Lu
1154e23e2d
add logs for volume creation
2021-08-10 13:04:25 -07:00
Chris Lu
057ef429ac
format
2021-08-10 12:33:29 -07:00
Chris Lu
b63b042afc
dedup keeps the largest replica
2021-08-10 12:30:41 -07:00
Chris Lu
0526db12e2
do not treat read only volumes differently
2021-08-10 11:37:12 -07:00
Chris Lu
e50a5b8e28
minor: print disk type
2021-08-10 11:10:09 -07:00
Chris Lu
db6275a0c8
print out balance ratio
2021-08-10 10:43:42 -07:00
Chris Lu
9d85569c55
ensure using local quorum consistency
2021-08-10 05:10:57 -07:00
Chris Lu
85832d02c0
wait for goroutines
2021-08-10 04:13:12 -07:00
Bl1tz23
1c94b3d013
merge master, resolve conflicts
2021-08-10 13:45:24 +03:00
Bl1tz23
e6e57db530
Add liveness\readiness probe for s3 api handler on /status path
2021-08-10 13:42:46 +03:00
Chris Lu
8ff6c9a0c6
output format
2021-08-10 03:25:18 -07:00
Chris Lu
48f448ee09
parallelize tier move
2021-08-10 03:08:29 -07:00
Chris Lu
69a6da7969
avoid fail on tail error
2021-08-10 02:50:28 -07:00
Chris Lu
18228f3044
fix help message
2021-08-10 02:48:41 -07:00
Chris Lu
69655ba8e5
mount: cache on reading remote storage
2021-08-09 22:11:57 -07:00
Chris Lu
a7012d9729
fix
2021-08-09 16:03:03 -07:00
Chris Lu
02f728cb54
fix bug with remote.uncache
2021-08-09 15:24:21 -07:00
Chris Lu
8d3e275735
remote: filer cache remote content on read
2021-08-09 15:16:45 -07:00
Chris Lu
9096f6f4f7
cache: set upper limit of chunk size
2021-08-09 15:08:53 -07:00
Chris Lu
402315f117
go fmt
2021-08-09 14:37:34 -07:00
Chris Lu
a6be2520c9
fix
2021-08-09 14:37:25 -07:00
Chris Lu
713c035a6e
shell: remote.cache remote.uncache
2021-08-09 14:35:18 -07:00
Chris Lu
8cfd487608
2.62
2021-08-08 23:33:12 -07:00
Chris Lu
734c980040
volume: support concurrent download data size limit
2021-08-08 23:25:16 -07:00
Chris Lu
96ce85f5ae
rename
2021-08-08 22:33:31 -07:00
Chris Lu
c5f38c365d
go fmt
2021-08-08 22:30:36 -07:00
Chris Lu
df85f7a1eb
adjust help message
2021-08-08 22:30:12 -07:00
Chris Lu
c0b12da4ef
shell: add filer.remote.unmount
2021-08-08 22:26:37 -07:00
Chris Lu
882a93dacd
fix tests
2021-08-08 17:56:26 -07:00
Chris Lu
7412ccdf88
write back remote entry to local entry after uploading to remote
2021-08-08 17:55:03 -07:00
Chris Lu
dcf614a8c3
skip if the remote entry update is because of internal managerial operations
2021-08-08 15:58:10 -07:00
Chris Lu
eed26af266
Merge branch 'master' into add_remote_storage
2021-08-08 15:48:04 -07:00
Chris Lu
4370a4db63
use int64 for volume count in case of negative overflow
2021-08-08 15:19:39 -07:00
Chris Lu
26c222f596
shell: volume.tier.move avoid moving all volumes to one destination
2021-08-08 15:12:39 -07:00
Chris Lu
13e45e1605
filer.remote.sync can work now
2021-08-08 01:21:42 -07:00
Chris Lu
8f5170c138
remove imports
2021-08-07 16:20:17 -07:00
Chris Lu
46b0cb8c86
Merge branch 'master' into add_remote_storage
2021-08-07 15:41:27 -07:00
Chris Lu
de730b079d
ChunkStreamReader implenents io.ReaderAt
2021-08-07 15:41:07 -07:00
Chris Lu
59732a0529
refactoring
2021-08-07 15:35:27 -07:00
Chris Lu
ecb234f75a
refactor
2021-08-07 14:46:23 -07:00
Chris Lu
270770d7d7
refactor
2021-08-07 14:18:53 -07:00
Chris Lu
679f800caa
Merge branch 'master' into add_remote_storage
2021-08-06 20:27:12 -07:00
Chris Lu
0c0f77e2ae
skip not found error on deletion
2021-08-06 19:35:47 -07:00
Chris Lu
8828f485c0
print volume deletion error
2021-08-06 19:30:22 -07:00
Chris Lu
d2b23f3d38
Merge pull request #2238 from combineads/change_default_mode
...
Change default permissions
2021-08-05 21:28:06 -07:00
Chris Lu
e23ca3845b
Merge pull request #2237 from combineads/add_mtime
...
Add crtime and mtime
2021-08-05 21:27:13 -07:00
Chris Lu
286e5dd375
Merge branch 'master' into add_remote_storage
2021-08-05 21:07:04 -07:00
Chris Lu
1e22166939
adjust error message
2021-08-05 21:06:55 -07:00
byunghwa.yun
f3dc909b21
Change default permissions
2021-08-06 12:24:35 +09:00
byunghwa.yun
e84fad9acf
Add crtime and mtime
2021-08-06 12:13:37 +09:00
Chris Lu
1a5d29520c
add tests
2021-08-05 14:49:24 -07:00
Chris Lu
6b743dbbf9
refactor client subscribe metadata
2021-08-04 16:25:46 -07:00
Chris Lu
b9ecf1e3a8
refacotring
2021-08-04 14:56:13 -07:00
Chris Lu
d84c311699
refactoring
2021-08-04 12:30:18 -07:00
Chris Lu
f6a9ad8001
fix tests
2021-08-04 00:31:06 -07:00
Chris Lu
42969c9c62
Merge branch 'master' into add_remote_storage
2021-08-03 21:20:05 -07:00
Chris Lu
8cf0c515bf
shell: volume.fix.replication retries even when there is no error #2235
...
fix https://github.com/chrislusf/seaweedfs/issues/2235
2021-08-03 09:32:55 -07:00
Chris Lu
d2ddf1dbdb
typo
2021-08-03 09:32:31 -07:00
Chris Lu
3afbf04007
add TLS grpc support for filer meta clients
2021-08-03 01:30:35 -07:00
Chris Lu
fe60c6ef9a
minor changes
2021-08-02 11:49:40 -07:00
byunghwa.yun
bdc7730fdb
Add autocomplete
2021-08-03 02:56:52 +09:00
Chris Lu
2ae9705442
adjust text
2021-08-01 22:55:19 -07:00
Chris Lu
3739717092
Revert "adds a test"
...
This reverts commit f690643b47
2021-08-01 22:54:45 -07:00
Chris Lu
f690643b47
adds a test
2021-08-01 22:53:50 -07:00
Chris Lu
6de786185d
volume.balance: balance read only volumes first
2021-08-01 22:32:50 -07:00
Chris Lu
3bb640b786
add -force option following #2228
2021-08-01 20:03:05 -07:00
byunghwa.yun
9e839cb5cc
Add force option in volume.deleteEmpty command
2021-08-02 11:37:20 +09:00
Chris Lu
56ee1d5ef1
2.61
2021-08-01 15:50:19 -07:00
Chris Lu
81b255df8b
Merge branch 'master' into add_remote_storage
2021-08-01 15:44:21 -07:00
Chris Lu
58bc3ecf47
add default quietFor value
2021-08-01 15:36:06 -07:00
Chris Lu
2ca1839d77
shell: add volume.deleteEmpty command
2021-08-01 15:33:45 -07:00
Chris Lu
89933c46d2
s3: skip hidden directories in /buckets folder
2021-08-01 12:28:08 -07:00
Chris Lu
1c7e404abe
remove buckets folder option
...
the related code still works for old deployments
2021-08-01 12:23:16 -07:00
Chris Lu
cb1dbd3135
refactor
2021-08-01 11:53:46 -07:00
Chris Lu
1161c70dab
Merge branch 'master' into add_remote_storage
2021-08-01 01:10:13 -07:00
Chris Lu
9cc84a910f
volume: deletion can skip volume size checking
...
fix https://github.com/chrislusf/seaweedfs/issues/2225
2021-08-01 00:32:51 -07:00
Chris Lu
767edd3c08
rename
2021-07-31 23:52:09 -07:00
Chris Lu
9df7d16791
read <- remote_storage
2021-07-31 22:39:38 -07:00
Chris Lu
f5a69a0e44
Merge branch 'master' into add_remote_storage
2021-07-31 09:39:48 -07:00
Chris Lu
6ba65c3382
customizable debug port
2021-07-31 09:18:41 -07:00
byunghwa.yun
5d4438a72f
Fix typo
2021-07-31 22:23:52 +09:00
Chris Lu
1ff8285d82
debug from any server
2021-07-31 02:13:21 -07:00
Chris Lu
0d6f45cb46
add debug option to generate full goroutine stack dump
2021-07-31 02:00:01 -07:00
Chris Lu
a3290faf17
shell command to calculate size / number of volumes in a collection
...
fix https://github.com/chrislusf/seaweedfs/issues/2224
2021-07-30 15:49:46 -07:00
Chris Lu
c6f992b2a3
remove dead code
2021-07-30 15:18:01 -07:00
Chris Lu
899963ac20
remote storage location changed to struct
2021-07-29 02:08:55 -07:00
Chris Lu
c090d6bb25
add ReadRemote(), add read remote setup when filer starts
2021-07-28 22:43:12 -07:00
Chris Lu
4deac06da0
log fix
2021-07-28 22:25:04 -07:00
divanikus
5c6270a93a
shell: ability to use wildcards for collections, all collections if ommited
2021-07-27 23:53:01 +03:00
Chris Lu
035b0bae29
refactor
2021-07-27 03:32:24 -07:00
Chris Lu
1752eeb538
remote.mount saves the mapping
2021-07-27 03:26:35 -07:00
Chris Lu
4b94b03d90
directory to remote storage mapping
2021-07-27 01:16:28 -07:00
Chris Lu
99b599aa8a
remote.mount
2021-07-26 22:53:44 -07:00
Chris Lu
5dede5d38d
2.60
2021-07-25 22:09:09 -07:00
Chris Lu
35f70c51b0
refactor
2021-07-25 03:37:37 -07:00
Chris Lu
72eec84167
shell: add fs.mkdir
2021-07-25 02:37:39 -07:00
byunghwa.yun
2b28a818f1
Fix mysql sql for batch delete
2021-07-25 11:06:14 +09:00
Chris Lu
10464f47d0
Merge pull request #2217 from combineads/add_webp_format
2021-07-24 09:42:41 -07:00
byunghwa.yun
7374b5b473
Add webp extension for resizing
2021-07-24 14:32:31 +09:00
byunghwa.yun
2595f269d1
Add the webp image type
2021-07-24 14:26:40 +09:00
Chris Lu
ac28611817
snowflake sequencer need an unique id
...
fix https://github.com/chrislusf/seaweedfs/issues/2213
2021-07-23 20:54:03 -07:00
Chris Lu
5c14da0f1e
filer.copy: fail early if assign request failed
...
fix https://github.com/chrislusf/seaweedfs/issues/2216
2021-07-23 20:01:43 -07:00
Chris Lu
4cc2165061
weed -h subcommand list is not sorted, makes discovery difficult
...
fix https://github.com/chrislusf/seaweedfs/issues/2214
2021-07-23 18:44:53 -07:00
Chris Lu
10fc478557
scaffold -config= should output to stdout
...
fix https://github.com/chrislusf/seaweedfs/issues/2212
2021-07-23 18:41:25 -07:00
Chris Lu
092932af8f
shell: faster bucket delete
...
avoid waiting for large object metadata
2021-07-23 11:25:46 -07:00
Chris Lu
bdb632fa62
Merge pull request #2211 from qieqieplus/filer-bloom-filter
...
add bloom filter for filer leveldb/rocksdb
2021-07-22 23:35:54 -07:00
qieqieplus
2b46df13f8
add bloom filter for filer leveldb/rocksdb
2021-07-23 14:05:59 -06:00
Chris Lu
60f5c0a2f5
fix security alert on github.com/dgrijalva/jwt-go
...
resolve https://github.com/chrislusf/seaweedfs/security/dependabot/go.mod/github.com%2Fdgrijalva%2Fjwt-go/open
2021-07-22 10:22:09 -07:00
Chris Lu
182288f860
filer: fix mysql, postgres batch delete error
2021-07-22 08:23:20 -07:00
Chris Lu
84d91f143f
mount: hide /etc folder also from mount
2021-07-22 00:40:16 -07:00
Chris Lu
7359193e97
go fmt
2021-07-21 14:38:12 -07:00
Chris Lu
cc32436d63
Merge pull request #2209 from combineads/fuse_option
...
Fix error handling
2021-07-21 12:37:20 -07:00
Chris Lu
67537ff05b
Merge pull request #2208 from combineads/add_rm_options
2021-07-21 11:08:43 -07:00
byunghwa.yun
e0c7708b03
Fix error handling
2021-07-21 22:48:04 +09:00
byunghwa.yun
41d559f54e
Add fs rm options
2021-07-21 21:10:36 +09:00
Chris Lu
70effac0d3
configure and store remote configurations
2021-07-21 02:24:34 -07:00
Chris Lu
3138805b33
Merge pull request #2200 from combineads/add_fs_rm
...
weed shell: fs.rm remove file or directory
2021-07-21 00:35:54 -07:00
Chris Lu
a125c8fbe2
Merge pull request #2204 from nivekuil/tokenpolicy
...
cassandra: use LocalQuorum for all queries
2021-07-20 15:37:31 -07:00
nivekuil
b9a67d46c5
cassandra: use LocalQuorum for all queries
...
This changes this filer store from eventual to strong consistency at the cost
of read performance.
2021-07-20 14:47:39 -07:00
Chris Lu
54c8bc8673
Merge branch 'remote_overlay'
2021-07-19 23:18:35 -07:00
Chris Lu
7ab389e7ec
optimization: improve random range query for large files
2021-07-19 23:07:22 -07:00
Chris Lu
93c37cfded
Update filechunk_manifest.go
2021-07-19 18:41:41 -07:00
byunghwa.yun
b23b307e08
Add fs rm
2021-07-20 08:49:28 +09:00
Chris Lu
b938df97a2
remove unused parameter
2021-07-19 02:59:12 -07:00
Chris Lu
450222dd64
add remote to filer.Entry and filer_pb entry, add RemoteConf
2021-07-19 02:47:27 -07:00
Chris Lu
8dc5def435
proto: add remote
2021-07-19 01:12:31 -07:00
Chris Lu
99155e4880
add -retry option for volume.fix.replication
...
fix https://github.com/chrislusf/seaweedfs/issues/2191
2021-07-16 12:13:46 -07:00
Chris Lu
2f209675ab
Added -retry option for volumeServer.evacuate
...
related to https://github.com/chrislusf/seaweedfs/issues/2191
2021-07-16 12:08:21 -07:00
Chris Lu
fb7a1be1c4
refactor
2021-07-16 11:47:37 -07:00
nivekuil
2faf96f002
cassandra: Use LocalOne instead of One consistency
2021-07-16 04:29:46 -07:00
Chris Lu
4be5b4ff5c
[volume.check.disk] skip errors and check next volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/2129
2021-07-16 02:03:32 -07:00
Chris Lu
e95166d739
S
2021-07-15 17:43:38 -07:00
Chris Lu
a45bbc0b75
2.59
2021-07-15 15:52:22 -07:00
Chris Lu
f0042f62dd
readable logs
2021-07-15 12:17:48 -07:00
Chris Lu
18c40686d9
s3: multipart upload miss data if file is chunked in 4MB
...
fix https://github.com/chrislusf/seaweedfs/issues/2195
2021-07-15 11:56:28 -07:00
Chris Lu
5a838dbe53
fix compilation
2021-07-13 11:40:21 -07:00
Chris Lu
6103649ffb
shell: volume.check.disk adds retries in case the volumes are just moved
...
related to https://github.com/chrislusf/seaweedfs/issues/2194
2021-07-13 11:19:56 -07:00
Chris Lu
49c66e88a0
volume: change all writes to fsync during graceful stopping
...
fix https://github.com/chrislusf/seaweedfs/issues/2193
2021-07-13 01:29:57 -07:00
Chris Lu
01adc567aa
shell: volume.fsck deletes a volume is the volume has only orphaned data.
...
fix https://github.com/chrislusf/seaweedfs/issues/2190
2021-07-12 11:22:00 -07:00
Chris Lu
297b41266b
2.58
2021-07-12 01:33:47 -07:00
Chris Lu
b194f91f47
add version to filer configuration response
2021-07-12 01:23:20 -07:00
bingoohuang
9e48bff8ee
fix typo in fs.configure help message text
2021-07-12 13:51:21 +08:00
Chris Lu
d013d6d968
shell: volume.fsck "reallyDeleteFromVolume" should send padded file ids to delete
...
fix https://github.com/chrislusf/seaweedfs/issues/2188
2021-07-10 23:16:06 -07:00
Chris Lu
3d624d1e16
rename
2021-07-09 03:19:21 -07:00
Chris Lu
ecce300964
s3 config read via grpc
2021-07-09 02:48:03 -07:00
Chris Lu
da7bd62822
/etc files are stored inside metadata store
2021-07-09 02:33:14 -07:00
Chris Lu
a6d73e0a66
Merge pull request #2185 from bingoohuang/master
...
show RemoteVolumes/EcVolumes only if it is not empty
2021-07-06 00:28:07 -07:00
bingoohuang
ed57a55eae
show RemoteVolumes/EcVolumes only if it is not empty
2021-07-06 15:20:18 +08:00
Chris Lu
44b50b2fdf
another fix related to 64 bit alignment
...
fix https://github.com/chrislusf/seaweedfs/issues/2177
2021-07-05 16:30:43 -07:00
Chris Lu
141388367e
Merge pull request #2183 from bingoohuang/master
...
extract embed html of master/volume/filer ui to separate files
2021-07-05 03:22:26 -07:00
bingoohuang
44a2538f67
extract embed html of master/volume/filer ui to separate files
2021-07-05 18:09:44 +08:00
Chris Lu
e2da647fa3
Merge pull request #2182 from qieqieplus/fix-event-notification
...
Subscribe meta data: sync empty notification with timestamp
2021-07-05 02:00:53 -07:00
qieqieplus
233103f6b2
sync empty notification with timestamp
2021-07-05 16:01:16 +08:00
bingoohuang
5dbbe19c8b
extract embed toml example to separate files
2021-07-05 11:16:49 +08:00
Chris Lu
5bcc77b46c
volume: default readMode to proxy
2021-07-03 15:55:56 -07:00
Chris Lu
f5fa0b08fd
2.57
2021-07-03 15:10:57 -07:00
Chris Lu
a024254ad7
logging
2021-07-03 14:51:01 -07:00
Chris Lu
d39b2689a5
S3 authorization: StreamingSigned enforces access control
...
fix https://github.com/chrislusf/seaweedfs/issues/2180
2021-07-03 14:50:53 -07:00
Chris Lu
fa0dab6029
mount: rename also recursively move file handles
...
related to https://github.com/chrislusf/seaweedfs/issues/2169
2021-07-03 02:59:35 -07:00
Chris Lu
8fe75692ee
volume: address "unaligned 64-bit atomic operation"
...
fix https://github.com/chrislusf/seaweedfs/issues/2177
2021-07-02 13:57:43 -07:00
Chris Lu
2cb8b31ea7
Merge pull request #2179 from nivekuil/tokenpolicy
...
Cassandra: Use TokenAwareHostPolicy with fallback to localDC by default
2021-07-02 13:51:53 -07:00
nivekuil
8425705643
Cassandra: Use TokenAwareHostPolicy by default with fallback
...
See https://pkg.go.dev/github.com/gocql/gocql#hdr-Data_center_awareness_and_query_routing
2021-07-02 13:50:01 -07:00
Chris Lu
d8bda0b229
locate the weed binary
2021-07-02 13:33:17 -07:00
danielflira
586e066897
fix weed fuse parameters parsing
2021-07-02 16:52:52 -03:00
lyg
1a4db87e19
fix s3 metadata error with multipart upload
2021-07-02 11:00:42 +08:00
Chris Lu
c6d4c16079
S3: add metadata with multipart upload
...
fix https://github.com/chrislusf/seaweedfs/issues/2173
2021-07-01 19:12:11 -07:00
Chris Lu
2420c60fc4
log reading adds delay between retries
2021-07-01 14:01:25 -07:00
Chris Lu
067eb15e70
remove debug messages
2021-07-01 01:24:07 -07:00
Chris Lu
b624090398
go fmt
2021-07-01 01:21:14 -07:00
Chris Lu
215b169562
mount: recursively rename locally
2021-07-01 01:19:31 -07:00
Chris Lu
c795183887
return node itself as directory handler
2021-07-01 01:19:31 -07:00
Chris Lu
30dbe98dff
Update Makefile
2021-07-01 01:19:31 -07:00
Chris Lu
07f20155fd
small optimization
2021-07-01 01:19:31 -07:00
danielflira
a66112c9d2
configurable fusermount path
2021-07-01 00:07:54 -03:00
Kevin Liu
253b4ed898
remote -> proxy in server to match volume flags
...
noticed this was missed in https://github.com/chrislusf/seaweedfs/pull/2168
2021-06-30 13:05:45 -07:00
zhangsong
20d33ae025
add proxy mode to read non-local volumes
2021-06-30 18:33:18 +08:00
zhangsong
7566782c2e
add proxy mode to read non-local volumes
2021-06-30 17:28:37 +08:00
陈杨文
6f683e6572
also failed on mips64
...
alpine support mips64, will try to build on next version
2021-06-30 04:36:00 +08:00
Chris Lu
41db292332
skip s390 ppc64le
...
due to https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/22703
2021-06-29 10:06:38 -07:00
Chris Lu
3668d10664
range query for mp4 video play
...
fix https://github.com/chrislusf/seaweedfs/issues/2156
2021-06-29 02:13:29 -07:00
Chris Lu
24e11d1e90
look back when adding to sorted values
...
look back when adding to sorted values, before adding it to overflow
2021-06-28 22:46:49 -07:00
Chris Lu
fc8dd58aea
volume: large_volume version has bug when using in memory index
...
fix https://github.com/chrislusf/seaweedfs/issues/2162
2021-06-28 15:48:07 -07:00
Chris Lu
a2979aa051
2.56
2021-06-27 23:33:45 -07:00
Chris Lu
08377fecb8
fixes
2021-06-27 23:32:57 -07:00
Chris Lu
17477b37d5
sleep before re-reading the messages
...
If there are no more metadata changes and the client disconnects, it would go into a busy loop without this fix.
2021-06-27 06:31:04 -07:00
Chris Lu
c764596f96
filer: slow metadata topic read may lose some change events
...
fix https://github.com/chrislusf/seaweedfs/issues/2117
2021-06-27 05:54:16 -07:00
Chris Lu
cc7714fdbe
logging changes to debug
2021-06-27 05:54:16 -07:00
thephoenixofthevoid
8d70ba2eaa
Fix: Loosing environment variables at StartProcess
2021-06-26 15:20:27 +03:00
Chris Lu
1cac2f2278
shell: volume.fsck checks missing chunks in filer
...
fix https://github.com/chrislusf/seaweedfs/issues/2154
2021-06-24 23:56:24 -07:00
Chris Lu
9dd09bbb33
refactor
2021-06-24 17:22:53 -07:00
Chris Lu
c2e0a75c1f
adjust logs
2021-06-24 12:46:00 -07:00
Chris Lu
78b1fb921c
adjust log level
2021-06-23 20:59:54 -07:00
Chris Lu
05af54ad10
2.55
...
trigger migrated travis build
2021-06-23 00:41:04 -07:00
Chris Lu
d4b0dcf442
Merge pull request #2147 from jonaz/process_metrics
...
Add process metrics of weed itself
2021-06-22 04:13:17 -07:00
Jonas Falck
829b195084
Add process metrics of weed itself
2021-06-22 13:09:42 +02:00
Chris Lu
d474ce6fe3
master: avoid repeated leader redirection
...
fix https://github.com/chrislusf/seaweedfs/issues/2146
2021-06-21 22:56:07 -07:00
danielflira
f1d207a0fe
start weed with mount in background
2021-06-20 02:48:46 -03:00
Chris Lu
b3eb4fecc7
2.54
2021-06-19 03:48:15 -07:00
Chris Lu
f24bb9e688
mount: fix for deletion stopped working since 2.53
...
fix https://github.com/chrislusf/seaweedfs/issues/2138
due to 4d0cbd2700
2021-06-19 03:46:39 -07:00
Chris Lu
88d52adfdd
remove unused fields
2021-06-18 15:35:22 -07:00
Chris Lu
56eb522b13
fix stats when a collection is deleted
2021-06-15 21:11:31 -07:00
Chris Lu
055374a50b
FUSE: skip flushing if file is deleted
...
related to https://github.com/chrislusf/seaweedfs/issues/2110
2021-06-15 12:45:23 -07:00
Konstantin Lebedev
6aa1a56ec8
avoid crashes Galera Cluster
...
https://github.com/chrislusf/seaweedfs/issues/2125
2021-06-15 18:12:39 +05:00
Chris Lu
ebe971da2e
Merge pull request #2127 from nagy/check-history-error
...
Check for history file errors as well
2021-06-15 03:29:21 -07:00
Chris Lu
7a81caa31e
2.53
2021-06-13 17:12:34 -07:00
Chris Lu
4d0cbd2700
skip cookie checking if from grpc api
2021-06-13 16:16:11 -07:00
Chris Lu
7225cb4ac5
add block and mutex profiling
2021-06-13 16:15:54 -07:00
Chris Lu
28a4a1f8d6
fix for mysql2 postgres2 on fast dropping buckets
2021-06-13 07:31:56 -07:00
Chris Lu
1e76fc994a
filer.copy: zero fileSize for directories
2021-06-13 07:09:06 -07:00
Chris Lu
ed6aa13520
minor
2021-06-12 18:36:25 -07:00
Chris Lu
ee6c67682c
minor
2021-06-12 02:52:41 -07:00
Chris Lu
9357911a95
remove all bucket metadata
...
fix https://github.com/chrislusf/seaweedfs/issues/2118
2021-06-10 23:37:54 -07:00
Chris Lu
b71c3cfba4
avoid possible empty object
2021-06-10 22:17:53 -07:00
Chris Lu
310e31424e
adjust the error output
...
fix https://github.com/chrislusf/seaweedfs/issues/2123
2021-06-10 21:55:13 -07:00
Chris Lu
8b382a8209
refactor
2021-06-10 21:50:21 -07:00
Chris Lu
33b87244ef
refactoring
2021-06-10 16:54:36 -07:00
Daniel Nagy
7ce3bee0c0
Check for history file errors as well
2021-06-08 11:14:49 +02:00
Daniel Nagy
0e3adde47f
Return correct exitcode when wheed upload fails
2021-06-07 23:38:14 +02:00
Chris Lu
76f24af79a
2.52
2021-06-07 12:13:23 -07:00
Chris Lu
5e6dfbc25f
locks for data racing
2021-06-07 12:04:50 -07:00
Chris Lu
452c6ef183
limits concurrent uploads for one file
2021-06-06 23:05:17 -07:00
Chris Lu
8295e2feb6
skip md5 checking for now because of race condition
2021-06-06 22:16:32 -07:00
Chris Lu
c8dea3dd89
2.51
2021-06-06 21:54:00 -07:00
Chris Lu
19caeb7b02
fix writing the small file
2021-06-06 20:57:03 -07:00
Chris Lu
bb45dea15a
filer: parallel data upload
2021-06-06 20:23:36 -07:00
Chris Lu
e00443a940
mount: adjust starting order
...
avoid possible nil wfs.Server
2021-06-06 20:22:42 -07:00
Chris Lu
44f1ba6894
refactor
2021-06-06 18:43:04 -07:00
Chris Lu
6c82326575
use bytes.Buffer to reduce memory allocation and gc
2021-06-06 13:42:36 -07:00
Chris Lu
9cba5cca0b
optionally disable concurrent upload limit
2021-06-06 13:13:33 -07:00
Chris Lu
21ad9a4ac2
filer: mongodb avoids E11000 duplicate key error collection: seaweedfs.filemeta index: directory_1_name_1 dup key
2021-06-06 13:12:01 -07:00
Chris Lu
e6ba2f9c37
verbose but cleaner logic to invalidate fuse cache
2021-06-05 13:09:37 -07:00
Chris Lu
fadc1febdb
FUSE: invalidate FUSE cached entries
...
fix https://github.com/chrislusf/seaweedfs/issues/2108
2021-06-05 02:23:07 -07:00
Chris Lu
ab606dec2a
filer: add path-specific option to enforce readonly
2021-06-04 01:03:41 -07:00
Chris Lu
0a5388744c
Merge pull request #2105 from Woellchen/current_and_parent_dot_directories
...
Return artificial . and .. directories
2021-06-02 12:54:44 -07:00
Patrick Schmidt
77100754e6
Return artificial . and .. directories
2021-06-02 21:28:02 +02:00
Chris Lu
62142ff1d2
Merge pull request #2104 from kmlebedev/sync_volume_on_close
...
avoid data loss after restarting a container with a volum server
2021-06-02 11:47:57 -07:00
Chris Lu
cc34475012
remove file handle locking for setattr
2021-06-02 11:44:12 -07:00
Konstantin Lebedev
99ef280c7c
avoid data loss after restarting a container with a volum server
2021-06-02 17:07:19 +05:00
Chris Lu
3db1642392
mount: skip persisting metadata if file is open
2021-06-02 03:40:08 -07:00
Chris Lu
556cc3a4ca
mount: avoid exception if disk cache is not initialized
...
related to https://github.com/chrislusf/seaweedfs/issues/2102
2021-05-31 16:42:55 -07:00
Chris Lu
615cb24ba6
shell: use tab print style for completion
2021-05-31 03:29:29 -07:00
Chris Lu
c5de97ddbe
keep alive for streaming connections
...
fix https://github.com/chrislusf/seaweedfs/issues/2096
related to https://github.com/chrislusf/seaweedfs/issues/1277
2021-05-31 01:26:24 -07:00
Chris Lu
2b60e2abb1
only disallow streaming signed when no auth enabled
...
fix https://github.com/chrislusf/seaweedfs/issues/2101
2021-05-31 01:03:04 -07:00
Chris Lu
4233ad3f07
2.50
2021-05-30 20:40:30 -07:00
Chris Lu
1456616a77
recreate grpc connections if too many errors
...
address https://github.com/chrislusf/seaweedfs/issues/2098
2021-05-30 00:07:43 -07:00
Chris Lu
fb8036385a
s3: save metadata during put-object
...
fix https://github.com/chrislusf/seaweedfs/issues/2092
2021-05-29 14:14:30 -07:00
Chris Lu
2d7b4e5bb6
filer.backup: escape colon from path on windows
...
fix https://github.com/chrislusf/seaweedfs/issues/2084
2021-05-29 06:45:27 -07:00
Chris Lu
ae185b997f
Merge pull request #2099 from danielflira/mount-helper
...
fix parameter multiple values
2021-05-28 21:57:15 -07:00
danielflira
849f36c1ac
fix parameter multiple values
2021-05-29 00:37:25 -03:00
Eugeniy Kozlov
6e8bd16819
freespace params fix
2021-05-28 14:19:24 +03:00
Chris Lu
4d55132c7d
scaffold add comments
2021-05-27 14:40:25 -07:00
Chris Lu
45bffc92a8
filer.backup: fix cloud sinks when updating entry
2021-05-26 14:53:11 -07:00
Chris Lu
921e0d5008
remove verbose log
2021-05-26 14:43:34 -07:00
Chris Lu
1a70cb9b63
local sink: write and update files
...
fix issues with https://github.com/chrislusf/seaweedfs/issues/2084
2021-05-26 14:42:21 -07:00
Chris Lu
e699d16c85
Merge pull request #2085 from danielflira/mount-helper
...
create fuse subcommand to use weed with mount
2021-05-26 12:19:40 -07:00
danielflira
84488ebb33
replace filer.remote and parts[1]
2021-05-26 12:07:36 -03:00
Chris Lu
ac9bf71544
use jquery 3.6.0
2021-05-26 00:21:11 -07:00
danielflira
877c192af2
create fuse subcommand to use weed with mount
2021-05-25 23:32:35 -03:00
Chris Lu
67dd094b35
sqlite: does not support non-linux/darwin/windows
2021-05-25 17:36:15 -07:00
Chris Lu
dce1f02c9e
filer.backup: backup to local directory optionally is incremental
...
fixed one issue with https://github.com/chrislusf/seaweedfs/issues/2084
2021-05-25 17:19:20 -07:00
Konstantin Lebedev
b612d5aebd
s3 test get w/ If-Match: bogus ETag
2021-05-24 16:59:44 +05:00
Konstantin Lebedev
69c768870b
- object write cache control
...
- object write expires
2021-05-24 15:43:55 +05:00
Konstantin Lebedev
84dce32a57
Merge branch 'master' into head_check_all_chunks
2021-05-24 12:28:19 +05:00
Konstantin Lebedev
0e404cd84b
Chunk download stats
2021-05-24 12:14:50 +05:00
Konstantin Lebedev
bb42633522
rm chunk Download stats
2021-05-24 12:08:12 +05:00
Chris Lu
064269bb57
filer: Support sqlite as filer meta store
2021-05-23 23:58:28 -07:00
Chris Lu
b70aeb7585
bucket list only directories
2021-05-23 10:36:22 -07:00
Chris Lu
42fb03a66e
2.49
2021-05-23 00:51:47 -07:00
Chris Lu
3d3fa43542
filer: re-create grpc connections if having transport error
...
fix https://github.com/chrislusf/seaweedfs/issues/2070
2021-05-22 17:46:53 -07:00
Chris Lu
2d0a0733ec
filer: bootstrap from peer filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1861
2021-05-22 03:17:02 -07:00
Chris Lu
e5a2bf1287
s3: deprecating filer.options.buckets_fsync
2021-05-22 00:24:23 -07:00
Chris Lu
431684798b
s3: add errors if requests are signed by no authentication is setup
...
fix https://github.com/chrislusf/seaweedfs/issues/2075
2021-05-21 14:08:47 -07:00
Chris Lu
2f136a04a1
organize a bit better for temp file directory
2021-05-21 13:02:18 -07:00
Konstantin Lebedev
3325b850cc
rm func CheckAllChunkViews
2021-05-21 16:05:59 +05:00
Konstantin Lebedev
26a4f34a57
del checks
...
url err logging
stats chunk fetch
2021-05-21 15:59:12 +05:00
