mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-26 10:47:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
9,697 Commits 32 Branches 296 Tags 153 MiB
440ebd7b6d
Commit Graph

6265 Commits

Author SHA1 Message Date
Chris Lu
d28f8bb88f
Merge pull request #3192 from guol-fnst/weed_update 
remove "weed" from UsageLine, or weed will not show usage of weed update
 2022-06-16 22:58:29 -07:00
guol-fnst
c7c20881f5 remove "weed" from UsageLine, or weed will not show usage of weed update 2022-06-17 12:47:04 +08:00
chrislu
625fd16a2e reduce upfront memory usage for low density volume 2022-06-16 15:39:29 -07:00
Chris Lu
5f8d6b5f13
Merge pull request #3184 from guol-fnst/weed_update 
Weed update
 2022-06-16 10:02:45 -07:00
guol-fnst
cd078471c4 fix path bug on windows 
fix usage
 2022-06-16 17:24:56 +08:00
chrislu
aaa9938353 Revert "adjust conditions" 
This reverts commit e024586ff1.
 2022-06-16 01:26:36 -07:00
guol-fnst
995e221e97 remove -output, add -dir,-name for more flexibility 2022-06-16 16:07:32 +08:00
chrislu
e024586ff1 adjust conditions 2022-06-15 23:19:39 -07:00
Chris Lu
82f3bcc65e
Merge pull request #3081 from paochiang/volume_upload_limit_fix 
exclude replication from the concurrentUploadLimitMB
 2022-06-15 23:16:44 -07:00
liubaojiang
df0ce31a2e add condition when inFlightUploadDataLimitCond signal 2022-06-16 14:07:11 +08:00
zzq09494
62f74f5e3c add bucket label to s3 prometheus metrics 2022-06-16 13:21:25 +08:00
guol-fnst
4bc8792643 support specific version of weed 2022-06-16 11:40:23 +08:00
guol-fnst
1dbb925d29 remove unused function 
remove log when weed is up to date

add check whether weed is full version
 2022-06-16 11:40:20 +08:00
liubaojiang
3076ac101e move vs.concurrentUploadLimit != 0 out of the lock 2022-06-16 09:58:44 +08:00
Chris Lu
097a12dfb4
Merge pull request #3187 from ningfdx/remote 2022-06-15 06:34:32 -07:00
Chris Lu
55b9f281d1
Merge pull request #3185 from kmlebedev/logging_err_process_range_req 2022-06-15 06:33:33 -07:00
石昌林
78b3728169 add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations 
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply

# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply

# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply

# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply

# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
 2022-06-15 21:07:55 +08:00
ningfd
338705f375 fix(wdclient): GetLocations return 2022-06-15 19:20:13 +08:00
Konstantin Lebedev
31a929f7f2 logging processRangeRequest errors 2022-06-15 15:36:43 +05:00
Chris Lu
4eeeb5f50d
Merge pull request #3180 from blacktear23/issue-3149 2022-06-15 00:55:20 -07:00
yulai.li
358c0c0b37 Fix: provide filer.ui.deleteDir options for server command 2022-06-15 15:53:31 +08:00
Guo Lei
3df84cabf6
Merge branch 'chrislusf:master' into weed_update 2022-06-15 15:28:40 +08:00
guol-fnst
695e8c7555 fix minor bug 2022-06-15 15:26:04 +08:00
zhihao.qu
4d0d1848c6 fix(filer.sync): modify clientName format : from -> to 2022-06-15 13:33:20 +08:00
yulai.li
b90d213a87 Change ui.deleteDir default to true 2022-06-15 11:37:37 +08:00
zhihao.qu
42d04c581b feat(filer.sync): add metricsServer in filer.sync. 
Metrics include:
(1) the offset of the filer.sync
(2) the last send timestamp of the filer subscription
 2022-06-15 11:33:18 +08:00
chrislu
d4ef06cdcf 3.11 2022-06-14 12:14:52 -07:00
chrislu
b347b2fb54 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-14 11:16:51 -07:00
chrislu
36d2756978 volume: avoid writing too much data for range requests in large files 
related to https://github.com/chrislusf/seaweedfs/issues/3178
 2022-06-14 11:16:47 -07:00
yulai.li
28cda854eb Add filer command line parameter to let Filer UI show/hide directory delete button 2022-06-15 00:30:49 +08:00
zhihao.qu
14d82c3dea feat(filer.sync): add offset to path. 2022-06-14 19:46:02 +08:00
guol-fnst
78fc9cdfa0 add "copied from https://github.com/restic/restic/tree/master/internal/selfupdate" 2022-06-14 19:10:54 +08:00
guol-fnst
322dbcf372 add update feature 2022-06-14 19:01:57 +08:00
Chris Lu
b22ca85fbb
Merge pull request #3169 from shichanglin5/fix_auth 
fix: When there is no access permission configured before startup, th…
 2022-06-12 23:40:42 -07:00
shichanglin5
f01dd27752 fix: When there is no access permission configured before startup, the authentication does not take effect after configuring the permission after startup 2022-06-13 13:23:26 +08:00
chrislu
e437ec5a68 3.10 2022-06-12 19:30:27 -07:00
chrislu
d12f431d98 collect volume server status 2022-06-12 11:56:23 -07:00
chrislu
444ac21050 go fmt 2022-06-11 09:51:11 -07:00
chrislu
a74e926063 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-11 09:51:02 -07:00
chrislu
61b8c9c361 remote object store gateway: disable tagging for backblaze 2022-06-11 09:50:59 -07:00
Chris Lu
0a8b3948db
Merge pull request #3165 from blacktear23/reload-bug-fix 2022-06-10 07:47:43 -07:00
yulai.li
83f8cdd5cb Bug Fix: force reload filer page when upload/delete/rename/create directory 2022-06-10 22:39:21 +08:00
guol-fnst
eabf4d4e64 fix typo 2022-06-10 22:03:11 +08:00
Chris Lu
36757d28a8
Merge pull request #3163 from kmlebedev/metric_replica_placement_mismatch 
stats master_replica_placement_mismatch
 2022-06-10 06:44:16 -07:00
chrislu
9f8b72a54d Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID" 
This reverts commit 37da689319, reversing
changes made to 00d53c34c4.
 2022-06-10 06:38:17 -07:00
Konstantin Lebedev
fb57aa431c stats master_replica_placement_mismatch 2022-06-10 15:30:40 +05:00
shichanglin5
a63559ee6c fix: fix command prompt information 2022-06-10 12:46:10 +08:00
Chris Lu
37da689319
Merge pull request #3159 from shichanglin5/_duplicateUUID 
perf: Optimized volume handling duplicateUUID logic to avoid quitting…
 2022-06-09 18:01:02 -07:00
shichanglin5
f5b0c04b14 perf: Optimized volume handling duplicateUUID logic to avoid quitting when volume is actualy normal 
Under normal circumstances, there will be no problems, but when the
master is debugged in the local environment, the volume client cannot
communicate with the master normally, so the sendHeartBeat logic is
restarted, and a new connection is created to report the heartbeat. If
the master has not cleared the uuid of the volume at this time, then The
master will respond to volume duplicateUUIDS, and the volume service
will exit, but in fact the uuid of the volume is not duplicated
 2022-06-09 20:41:16 +08:00
zhihao.qu
cd5cca36a4 feat(filer.sync): add fromTsMs. Extract signature from doSubscribeFilerMetaChanges 2022-06-09 10:53:19 +08:00
Chris Lu
4a5135961f
Merge pull request #3154 from kmlebedev/s3_test_invalid_range 2022-06-08 01:45:40 -07:00
Konstantin Lebedev
49f058da85 fix s3 tests: 
ranged_request_invalid_range
ranged_request_empty_object
 2022-06-08 13:25:13 +05:00
chrislu
8a1166c021 auto remove previous leftover socket 2022-06-08 00:32:09 -07:00
chrislu
633a4aed18 filer: fix customizable local socket file name 
fix https://github.com/chrislusf/seaweedfs/issues/3147
 2022-06-08 00:15:28 -07:00
chrislu
acbc0286e2 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-07 22:33:38 -07:00
chrislu
ebd6097e6f fix: sequence read rate is slow when use filer.ChunkReadAt.ReadAt #3074 
related to https://github.com/chrislusf/seaweedfs/pull/3074
 2022-06-07 22:33:35 -07:00
Chris Lu
f43ec9f363
Merge pull request #3130 from kmlebedev/fix_rm_parent_dir_via_nextcloud 2022-06-07 08:56:55 -07:00
Konstantin Lebedev
c07820178f fix s3 tests 
bucket_list_delimiter_prefix
bucket_list_delimiter_prefix_underscore
bucket_list_delimiter_prefix_ends_with_delimiter
 2022-06-07 14:43:10 +05:00
chrislu
78be3b0f6c fix compilation 2022-06-06 10:09:21 -07:00
chrislu
4fd5f96598 filer: remove replication, collection, disk_type info from entry metadata 
these metadata can change and are not used
 2022-06-06 00:39:35 -07:00
chrislu
cbf46de5f4 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-06 00:15:40 -07:00
chrislu
8402af0882 filer: detect ttl based on fs configure setting 2022-06-06 00:15:37 -07:00
ningfd
f32142f6f5 add disableXAttr in mount option 2022-06-06 14:09:01 +08:00
chrislu
4a046e4de7 3.09 2022-06-05 19:00:24 -07:00
chrislu
11f1e7996b fix test 2022-06-05 18:50:41 -07:00
chrislu
d65bb2c6df mount: file handle locks entry better 
related to https://github.com/chrislusf/seaweedfs/issues/2952
 2022-06-05 18:15:06 -07:00
chrislu
746092a60b add a comment 2022-06-05 16:44:07 -07:00
chrislu
056c480eb0 volume: checksum remove the hashing step in Value() 2022-06-05 15:24:02 -07:00
chrislu
ecef844dfc stream read large files 2022-06-05 11:54:04 -07:00
chrislu
48cca4e54f remove unused logic 2022-06-04 19:11:06 -07:00
chrislu
685643d506 needle read into a writer 2022-06-04 19:10:52 -07:00
chrislu
663bc5dc23 add needle reading in chunks 2022-06-04 18:15:39 -07:00
chrislu
18b29f709c refactoring, split file 2022-06-03 23:31:14 -07:00
chrislu
39c3cb04af Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-03 08:45:32 -07:00
chrislu
f97acdd489 volume.fix.replication fix retry logic 
fix https://github.com/chrislusf/seaweedfs/issues/3136
 2022-06-03 08:45:29 -07:00
creeew
02ae102731 fix filer.sync missing source srv uploaded files to target when target down 2022-06-02 01:28:47 +08:00
Konstantin Lebedev
b86628f85d fix show empty dir 
https://github.com/chrislusf/seaweedfs/issues/3086
 2022-06-01 20:10:52 +05:00
chrislu
6793bc853c help message when in simulation mode 2022-05-31 14:48:46 -07:00
chrislu
89948a373b fix error reporting for "Need to a hdd disk type" 
related to https://github.com/chrislusf/seaweedfs/issues/3128
 2022-05-31 12:43:55 -07:00
chrislu
8a49240d64 3.08 2022-05-31 11:57:41 -07:00
chrislu
376b005ad3 fix wrong logic 2022-05-31 11:55:58 -07:00
chrislu
fbd99d53c1 3.07 2022-05-31 00:36:24 -07:00
chrislu
27732ecfa4 move s3 related constants from package http to s3_constants 2022-05-30 22:57:41 -07:00
chrislu
f4a6da6cb2 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-30 22:47:29 -07:00
chrislu
596c3860ca use final destination to resolve fs configuration 
related to https://github.com/chrislusf/seaweedfs/issues/3075
 2022-05-30 22:47:26 -07:00
Chris Lu
ca01ce0524
Merge pull request #3122 from chrislusf/filer-sync-with-peers 
Filer bootstrap from peers
 2022-05-30 21:38:31 -07:00
chrislu
6adc42147f fresh filer store bootstrap from the oldest peer 2022-05-30 21:27:48 -07:00
chrislu
490e0850bf filer.meta.tail add one example of using untilTimeAgo 2022-05-30 20:24:59 -07:00
liubaojiang
f0ee3e6f21 reduce the scope of inFlightUploadDataLimitCond lock 2022-05-31 09:49:07 +08:00
chrislu
c59068d0f3 refactor 2022-05-30 16:28:36 -07:00
chrislu
682382648e collect cluster node start time 2022-05-30 16:23:52 -07:00
chrislu
968ca95b49 filer.meta.tail: support untilTimeAgo for a range 2022-05-30 16:16:23 -07:00
chrislu
f214dfb1f5 stop when in memory log is done 2022-05-30 15:25:21 -07:00
chrislu
aece35a64f stop when on disk log is done 2022-05-30 15:20:51 -07:00
chrislu
a2b101a737 subscribe metadata between a range 2022-05-30 15:04:19 -07:00
chrislu
8902fa6ff6 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-30 13:28:32 -07:00
Chris Lu
730c9cf601
Merge pull request #3114 from kmlebedev/nextcloud 
Create folders in s3 via nextcloud
 2022-05-30 13:10:10 -07:00
chrislu
5b9347c938 typo 2022-05-30 12:37:41 -07:00
Konstantin Lebedev
af562e1a20 rm log info 2022-05-30 23:26:26 +05:00
Konstantin Lebedev
62e5e3822c list self dir 
https://github.com/chrislusf/seaweedfs/issues/3086
 2022-05-30 18:09:08 +05:00
Konstantin Lebedev
26f3646961 show fake dirs 
https://github.com/chrislusf/seaweedfs/issues/3086
 2022-05-30 15:03:27 +05:00
chrislu
b8f3db0d46 s3: keep auth enabled in case identities are set to empty 
fix https://github.com/chrislusf/seaweedfs/issues/3084
 2022-05-29 19:00:03 -07:00
chrislu
81d6159290 volume: report error if a volume has nil data backend 
fix https://github.com/chrislusf/seaweedfs/issues/3105
 2022-05-29 16:59:30 -07:00
chrislu
ca16fbf0ef shell: cluster.ps display filer group 2022-05-29 16:37:14 -07:00
chrislu
9219664505 reset sync offset if peer filer resets 
fix https://github.com/chrislusf/seaweedfs/issues/3089
 2022-05-29 13:11:24 -07:00
Konstantin Lebedev
54356211c2 list empty folder 2022-05-27 13:17:18 +05:00
Kaiwalya Joshi
4598581741
fix: Minor spelling fixes on FilerStorePathTranslator. 2022-05-25 01:11:02 -07:00
chrislu
5b8b022985 remove unused parameter 2022-05-24 18:52:04 -07:00
Robert Coelho
0e6e72d462 mount: ReadDir return EIO on EnsureVisited err 2022-05-24 13:02:59 -04:00
Robert Coelho
1fabbe8a25 mount: cleanup ReadDir rewrite's branches to not assume offsets 2022-05-24 12:48:32 -04:00
Robert Coelho
cb422d96f7 mount: rewrite ReadDir to respect input.Offset to fix partial results 2022-05-24 12:16:58 -04:00
Robert Coelho
05602167f4 mount: do not mark meta_cache when there is an err 2022-05-24 12:13:15 -04:00
chrislu
7e25a2d416 reduce busy waiting when reading metadata logs 2022-05-24 00:23:53 -07:00
chrislu
2f846777bb 3.06 2022-05-23 01:21:48 -07:00
chrislu
a5cdcc7e63 fix reading files larger than 20GB due to manifest resolving bug 2022-05-23 01:16:10 -07:00
chrislu
59cc3c0bfd fix reading files larger than 20GB due to manifest resolving bug 
this is introduced since Mar 08 2022
 2022-05-23 01:14:56 -07:00
chrislu
acc318e12b 3.05 2022-05-22 21:43:59 -07:00
liubaojiang
076e48a676 add inflight upload data wait timeout 2022-05-21 10:38:08 +08:00
liubaojiang
71b2e6223e add inFlightUploadDataLimitCond signal 2022-05-20 15:27:06 +08:00
liubaojiang
1a41691b4c exclude replication from the concurrentUploadLimitMB 2022-05-20 14:40:05 +08:00
Chris Lu
cb64983c6b
Merge pull request #3079 from ztinpn/hotfix_fsync 2022-05-19 10:33:42 -07:00
tianzhang
66747ee9c9 hotfix_fsync fix fsync 2022-05-19 23:17:17 +08:00
Chris Lu
8efe55f981
Merge pull request #3059 from guol-fnst/avoid_dup_vol 
avoid loading duplicated volume directory
 2022-05-17 23:48:57 -07:00
guol-fnst
8e2426a879 revert changes of pb file 
orignal protoc-gen-go-grpc verion is 1.1.0
 2022-05-18 11:52:14 +08:00
guol-fnst
b12944f9c6 fix naming convention 
notify volume server of duplicate directoris
improve searching efficiency
 2022-05-17 15:41:49 +08:00
guol-fnst
076595fbdd just exit in case of duplicated volume directories were loaded 2022-05-17 15:41:49 +08:00
guol-fnst
8fab39e775 rename UUID file 
fix typo
move locationUUID  into DiskLocation
 2022-05-17 15:41:47 +08:00
Chris Lu
72e7dcde51
Merge pull request #3060 from natmaka/master 2022-05-16 07:42:58 -07:00
Nat Makarevitch
b72f0634d7
minor (typos...), done while reading around 2022-05-16 22:11:33 +08:00
guol-fnst
de6aa9cce8 avoid duplicated volume directory 2022-05-16 19:33:51 +08:00
Konstantin Lebedev
44f53ceda6 fix collectionIsMismatch charset 2022-05-16 13:23:23 +05:00
Konstantin Lebedev
10d435f2c2 fix skip loop 2022-05-16 13:16:27 +05:00
Konstantin Lebedev
279053572c avoid delete volume replica if collection mismatch 2022-05-16 13:07:05 +05:00
chrislu
9ff0d99002 3.04 2022-05-15 21:32:21 -07:00
chrislu
866fed1eab avoid grpc name conflict 
fix https://github.com/chrislusf/seaweedfs/issues/3055
 2022-05-15 21:31:29 -07:00
chrislu
cd2d10118c 3.03 2022-05-15 18:17:07 -07:00
chrislu
271b5aed96 shell: volume.fsck add a note for -reallyDeleteFromVolume option 2022-05-15 11:07:04 -07:00
chrislu
4076d74813 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-15 00:43:39 -07:00
chrislu
29198720f2 s3: add grpc server to accept configuration changes 2022-05-15 00:43:37 -07:00
Chris Lu
ea4918394c
Merge pull request #3047 from shichanglin5/master 
fix the problem of metadata and tagging loss when files are copied
 2022-05-13 22:52:48 -07:00
shichanglin5
688d55488c test(s3api_object_copy_handlers_test.go): some unit tests have been added to the processMetadata & processMetadataBytes methods of s3api_object_copy_handlers.go 2022-05-14 10:40:29 +08:00
chrislu
735038b2c1 backup do not need to use preallocation 
fix https://github.com/chrislusf/seaweedfs/issues/3044
 2022-05-13 13:46:52 -07:00
shichanglin5
1166dead00 fix the problem of metadata and tagging loss when files are copied 
by adding processing of metadata and tagging in s3 api CopyObject (judging whether to copy or overwrite according to the directive header)
 2022-05-13 19:46:20 +08:00
Konstantin Lebedev
d12a423aa0 refactor 
add some tests

https://github.com/chrislusf/seaweedfs/pull/2996
 2022-05-12 12:23:02 +05:00
Konstantin Lebedev
f523aed3c9 fixes from comments 
https://github.com/chrislusf/seaweedfs/pull/2996
 2022-05-11 21:50:09 +05:00
chrislu
bff1ccc1de fix compilation 2022-05-11 00:52:15 -07:00
chrislu
76da688176 adjust log level 2022-05-11 00:45:19 -07:00
chrislu
3e20336bde fix atomic add int64 
fix https://github.com/chrislusf/seaweedfs/issues/3038#issuecomment-1123269831
 2022-05-11 00:44:35 -07:00
chrislu
a85720f0f6 remove deprecated function 2022-05-10 00:02:23 -07:00
chrislu
25559eeb92 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-10 00:01:46 -07:00
chrislu
791c29ae93 remove deprecated function 
WithQueryCachePolicyKeepInCache is deprecated and already default behavior
 2022-05-10 00:01:43 -07:00
qiang.zeng
06cb337b82 fix typo 2022-05-09 14:21:16 +08:00
chrislu
a4ca3ed1f0 3.02 2022-05-08 22:28:32 -07:00
chrislu
8bc16f1149 rename 2022-05-06 23:38:22 -07:00
chrislu
139e039c44 filer.sync: pass attributes for mount 
fix https://github.com/chrislusf/seaweedfs/issues/3012
 2022-05-06 03:54:12 -07:00
chrislu
10207a48a4 how to setup local test env 2022-05-06 02:25:56 -07:00
chrislu
db33b03735 add comments 2022-05-06 02:19:47 -07:00
chrislu
e350ba8896 larger test range 2022-05-06 02:17:28 -07:00
chrislu
71e0bcc5d5 a bit optimization 2022-05-06 02:11:58 -07:00
chrislu
b2a43dd044 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-06 01:51:28 -07:00
chrislu
d5fe5d12b7 filer: etcd fix pagination 
fix https://github.com/chrislusf/seaweedfs/issues/3016
 2022-05-06 01:51:19 -07:00
Chris Lu
a69bf53405
Merge pull request #2996 from kmlebedev/ydb 
ydb filer store
 2022-05-05 15:34:37 -07:00
Konstantin Lebedev
e6d2cb5911 ydb add empty doc file 2022-05-05 11:12:14 +05:00
chrislu
ef6c6c450e avoid fatal error if port is already in use 2022-05-04 14:55:14 -07:00
chrislu
f7366a9668 skip unix socket mode for windows 
fix https://github.com/chrislusf/seaweedfs/issues/3013
 2022-05-04 10:14:34 -07:00
Konstantin Lebedev
f58adaab25 build tag ydb since the growth of the binary is 5 mb 2022-05-03 23:20:09 +05:00
Konstantin Lebedev
f127b326bf add options to scaffold 2022-05-03 22:54:31 +05:00
Konstantin Lebedev
7b544576af refactor 2022-05-03 20:16:00 +05:00
chrislu
f17cd0d5cd return false if not found 
fix https://github.com/chrislusf/seaweedfs/issues/3011
 2022-05-03 07:18:34 -07:00
chrislu
b201edb9df fix wrong assignment 2022-05-03 07:12:33 -07:00
Konstantin Lebedev
04d5dff6bb add support native ttl 2022-05-03 17:52:23 +05:00
Konstantin Lebedev
cb3c7a3cdb enable query cache policy instead of prepare 2022-05-03 16:03:10 +05:00
Konstantin Lebedev
8342f651f3 fix scanNamed 2022-05-03 15:18:28 +05:00
chrislu
9271866d1e fix segmentation violation 
fix https://github.com/chrislusf/seaweedfs/issues/3000
 2022-05-02 23:56:11 -07:00
Konstantin Lebedev
2d27afb515 Merge branch 'new_master' into ydb 2022-05-03 10:47:39 +05:00
chrislu
76148ce0f7 use storage_class for backend configuration 
fix https://github.com/chrislusf/seaweedfs/issues/3008
 2022-05-02 12:20:01 -07:00
Konstantin Lebedev
d8925b4e83 Merge branch 'new_master' into ydb 
# Conflicts:
#	go.mod
#	go.sum
 2022-05-03 00:13:57 +05:00
Konstantin Lebedev
7640e650e5 fix queries 2022-05-03 00:11:37 +05:00
Konstantin Lebedev
319d300d48 fix createDB 2022-05-02 22:23:07 +05:00
Konstantin Lebedev
0dc44dda63 ydb do Tx or DB 2022-05-02 15:33:29 +05:00
Konstantin Lebedev
a3e48831f3 ydb BucketAware interface 2022-05-02 12:42:20 +05:00
chrislu
73961e24d8 3.01 2022-05-01 23:33:45 -07:00
chrislu
00c1dfec4f go fmt 2022-05-01 23:16:29 -07:00
chrislu
ce1e8e58ae fix test 2022-05-01 23:15:58 -07:00
chrislu
2c0fdf3693 fix tests 2022-05-01 23:14:04 -07:00
chrislu
94635e9b5c filer: add filer group 2022-05-01 21:59:16 -07:00
Konstantin Lebedev
50c4f62ed4 ydb-go-sdk move to v3 2022-05-02 02:07:47 +05:00
Konstantin Lebedev
21033ff4c3 refactor use const CountEntryChunksForGzip 2022-05-01 22:28:55 +05:00
Konstantin Lebedev
ec0ed41e37 ydb kv interface 2022-05-01 21:20:37 +05:00
chrislu
4bd6bea429 fix test 2022-05-01 09:02:01 -07:00
Konstantin Lebedev
6a052f6ff2 ydb Sql interface 2022-05-01 20:33:03 +05:00
Konstantin Lebedev
275c3bb19c ydb initial 
https://github.com/chrislusf/seaweedfs/issues/2942
 2022-05-01 17:28:58 +05:00
chrislu
5d081d5c3c skip if already copied to remote 
fix https://github.com/chrislusf/seaweedfs/issues/2993
 2022-04-30 20:03:04 -07:00
chrislu
1aae7a3f1b volume.tier.upload progress starts negative #2992 
fix https://github.com/chrislusf/seaweedfs/issues/2992
 2022-04-30 18:10:01 -07:00
chrislu
ea8a7a5584 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-04-30 17:36:45 -07:00
chrislu
192983b464 s3 backend support customizing storage class 2022-04-30 17:36:40 -07:00
Konstantin Lebedev
03aec9ce7e avoid panic if hashicorp raft not initialized 2022-04-30 13:56:35 +05:00
guosj
1a5de36257 fix return value 2022-04-29 11:19:36 +08:00
Chris Lu
4e97b6146a
Merge pull request #2982 from kmlebedev/issues/2981 
avoid empty listMultipartUploads response
 2022-04-27 17:21:44 -07:00
chrislu
b2a6111090 skip ipv6 all interfaces and localhost 
fix https://github.com/chrislusf/seaweedfs/issues/2983
 2022-04-27 17:18:09 -07:00
Konstantin Lebedev
306cf70c4a avoid empty listMultipartUploads response 2022-04-27 19:27:44 +05:00
chrislu
70e5a1b632 volume close should wait for committing compaction 2022-04-26 23:34:05 -07:00
Chris Lu
ffe028f8d0
Merge pull request #2974 from kmlebedev/wait_volume_closed_compression 
waite volume being closed during compression idx
 2022-04-26 23:29:22 -07:00
chrislu
37ab8909b0 use two flags: v.isCompacting and v.isCommitCompacting 2022-04-26 23:28:34 -07:00
Chris Lu
b557faf175
Merge pull request #2967 from kmlebedev/vl_chk_disk_vlid 
volume.check.disk add param volumeId
 2022-04-26 22:09:06 -07:00
Chris Lu
eb49bd5638
Merge pull request #2970 from kmlebedev/vlm_fsck_forcePurging 
delete missing data from volumes in one replica
 2022-04-26 22:08:34 -07:00
chrislu
94f824e1ce volume: sync to disk before copying volume files 
address https://github.com/chrislusf/seaweedfs/issues/2976
 2022-04-26 13:03:43 -07:00
Konstantin Lebedev
7315d1d039 waite volume being closed during compression idx 2022-04-26 13:40:42 +05:00
Konstantin Lebedev
d4343ab7da forcePurging desc 2022-04-25 23:11:56 +05:00
Konstantin Lebedev
ae56b2c00f change forcePurging to a pointer 2022-04-25 23:10:01 +05:00
Konstantin Lebedev
6d2fda27d2 delete missing data from volumes in one replica 2022-04-25 22:59:46 +05:00
Konstantin Lebedev
67476d830a volume.check.disk add param volumeId 2022-04-25 18:27:14 +05:00
chrislu
2aef5b8b32 3.00 2022-04-24 21:46:03 -07:00
Chris Lu
184e982898
Merge pull request #2957 from guo-sj/handle_implicit_username 
Handle implicit username
 2022-04-24 09:07:48 -07:00
guosj
8fd7b24b80 remove redundant logs & add unit test 2022-04-24 16:09:10 +08:00
guol-fnst
8b2552f944 fix http response error code 2022-04-24 11:07:27 +08:00
guosj
09befe871f Merge branch 'master' into handle_implicit_username 2022-04-23 09:48:25 +08:00
guosj
893a0eea57 add more checks and comments 2022-04-22 22:47:11 +08:00
chrislu
ad01c63b84 conditionally skip hdfs related code 2022-04-21 01:43:01 -07:00
chrislu
4aa39ef33f conditionally compile sqlite 2022-04-21 01:22:38 -07:00
chrislu
3885374edf conditionally build elastic, gocdk to reduce binary size 2022-04-21 01:10:46 -07:00
chrislu
6994e88192 fix typo 2022-04-21 00:02:18 -07:00
chrislu
cb1e7aa27d fix build 2022-04-20 16:21:56 -07:00
a
1d6a9e66b6 Merge branch 'master' into a 2022-04-20 14:01:42 -07:00
chrislu
bdc4d67de8 Revert "Merge pull request #2944 from guo-sj/handle_implicit_username" 
This reverts commit bc96e73b8d, reversing
changes made to 88a669dd19.
 2022-04-20 09:40:41 -07:00
chrislu
3aa4dc6ad5 Revert "fix index out of range" 
This reverts commit 4792e07228.
 2022-04-20 09:40:24 -07:00
chrislu
4792e07228 fix index out of range 2022-04-20 09:32:17 -07:00
chrislu
f3fdfb07b8 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-04-19 22:58:59 -07:00
chrislu
8e9ad7db5a fix bug deleting volume or unmount volume if a volume server has multiple directories 2022-04-19 22:58:57 -07:00
Chris Lu
bc96e73b8d
Merge pull request #2944 from guo-sj/handle_implicit_username 
handle implicit username
 2022-04-19 21:42:21 -07:00
guosj
a23629b2fe handle implicit username 2022-04-20 10:12:33 +08:00
Chris Lu
88a669dd19
Merge pull request #2858 from kmlebedev/fsck_replicas 
fsck replicas
 2022-04-19 00:17:43 -07:00
Chris Lu
f76c318a1f
Merge pull request #2940 from kmlebedev/volume.list.grep 
volume.list show only readonly, collectionPattern and volumeId
 2022-04-19 00:17:32 -07:00
Konstantin Lebedev
2364fab927 volume.list show only readonly, collectionPattern and volumeId 2022-04-19 11:44:41 +05:00
guosj
94ad557d97 add missing return 2022-04-19 14:16:32 +08:00
Konstantin Lebedev
36c5a59ed8 add help 2022-04-18 19:36:14 +05:00
Konstantin Lebedev
1e35b4929f shell vacuum volume by collection and volume id 2022-04-18 18:40:58 +05:00
Chris Lu
97c6e0efb1
Merge pull request #2931 from kmlebedev/fix_nil_dataBackend 
avoid invalid memory address or nil pointer dereference
 2022-04-18 00:13:26 -07:00
Konstantin Lebedev
9438738693 avoid invalid memory address or nil pointer dereference 2022-04-18 12:10:22 +05:00
chrislu
a69f0b62c2 start filer local socket only on non windows 
fix https://github.com/chrislusf/seaweedfs/issues/2930
 2022-04-18 00:08:31 -07:00
chrislu
9ae66f55fc 2.99 2022-04-17 23:01:42 -07:00
chrislu
4524ceae6b less verbose logs 
fix https://github.com/chrislusf/seaweedfs/discussions/2925#discussioncomment-2584085
 2022-04-17 22:55:26 -07:00
leyou240
89eb87c1d1
Merge branch 'master' into slices.SortFunc 2022-04-18 10:39:29 +08:00
justin
3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
chrislu
b597baf488 mount: set file write time correct to work well with rsync 
fix https://github.com/chrislusf/seaweedfs/issues/2867
 2022-04-16 22:56:45 -07:00
chrislu
460d56d283 shell: cluster.check prints out clock delta and network latency 2022-04-16 13:24:17 -07:00
chrislu
b4be56bb3b add timing info during ping operation 2022-04-16 12:45:49 -07:00
chrislu
1384529eb7 Fix filer.backup deletes files in backup folder in incremental mode 
fix https://github.com/chrislusf/seaweedfs/issues/2919
 2022-04-14 13:35:01 -07:00
a
846858fb43 merge master 2022-04-12 14:42:29 -07:00
chrislu
1f03fcccb1 fix nil in cluster_check shell command 
fix https://github.com/chrislusf/seaweedfs/issues/2905
 2022-04-12 08:47:27 -07:00
Chris Lu
dd13764a5c
Merge pull request #2900 from kmlebedev/fix_cleanupUploads 
avoid breaking loop in cleanupUploads if error is empty
 2022-04-12 02:03:31 -07:00
Konstantin Lebedev
76b1c5ce67 cleanupUploads use jwt token 2022-04-12 13:46:11 +05:00
Chris Lu
72db181d68
Merge pull request #2901 from guol-fnst/fix_multiupload 
check object name and uploadID when processing  multiupload
 2022-04-11 23:38:12 -07:00
guol-fnst
44d810d163 rename functions and remove uncessary check 2022-04-12 13:40:53 +08:00
guol-fnst
180aa88a92 check uploadid using object name hash string 2022-04-12 11:04:38 +08:00
guol-fnst
2232cfb5b7 Check object name and uploadID when processing multipart uploading 2022-04-11 19:53:44 +08:00
Konstantin Lebedev
2dfbe210e0 avoid breaking loop in cleanupUploads if error is empty 2022-04-11 11:16:20 +05:00
Chris Lu
a87f57e47c
Merge pull request #2868 from kmlebedev/hashicorp_raft 
hashicorp raft
 2022-04-10 23:00:05 -07:00
Konstantin Lebedev
931cb9e581 use "cluster.raft.{ps,add,remove}" 2022-04-11 10:50:01 +05:00
Konstantin Lebedev
d2fe0fae33 use the full ServerAddress as the identifier 2022-04-11 10:44:44 +05:00
chrislu
c6ec5269f4 2.98 2022-04-10 18:56:40 -07:00
chrislu
261ba0208e lock-protected setting entry 2022-04-09 22:52:59 -07:00
Chris Lu
9f4ecd50ac
Merge pull request #2889 from blacktear23/filer-ui-bootstrap-style 
Change filer UI icon from picture to icon font.
 2022-04-08 00:50:32 -07:00
yulai.li
ceb058ddd9 Change filer UI icon from picture to icon font. 2022-04-08 15:07:56 +08:00
chrislu
7f26676f57 adjust td width 2022-04-07 22:34:32 -07:00
yulai.li
dff63e9024 Make operation buttons show when table row hovered 2022-04-08 13:10:12 +08:00
chrislu
4dba102137 use icon instead of button text 2022-04-07 11:15:01 -07:00
Chris Lu
bbc95dac47
Merge pull request #2885 from binbinshi/master 
fix: master lose some volumes
 2022-04-07 10:16:19 -07:00
Konstantin Lebedev
104ea7b029 master ui add raft stats and Max Volume Id 2022-04-07 20:52:01 +05:00
a
41d396edc4 Merge branch 'master' into a 2022-04-07 08:22:59 -07:00
Konstantin Lebedev
a2fdb3e277 fix master ui 2022-04-07 19:37:40 +05:00
Konstantin Lebedev
35bc67f030 fix conflicts 2022-04-07 18:52:13 +05:00
Konstantin Lebedev
f5246b748d Merge branch 'new_master' into hashicorp_raft 
# Conflicts:
#	weed/pb/master_pb/master.pb.go
 2022-04-07 18:50:27 +05:00
yulai.li
11a9f993a8 Make filer-ui bootstrap style 2022-04-07 20:17:00 +08:00
Konstantin Lebedev
7ff248d5cd refactor OnPeerUpdate 2022-04-07 16:23:22 +05:00
Konstantin Lebedev
85d80fd36d fix removing old raft server 2022-04-07 15:31:37 +05:00
chrislu
abe3cc6df2 filer UI touch up 2022-04-07 01:25:55 -07:00
Chris Lu
869d23b099
Merge pull request #2884 from blacktear23/filer-ui-new-features 
Filer UI new features
 2022-04-07 00:58:56 -07:00
chrislu
86ed27f602 mount: remove leftover socket file 2022-04-07 00:33:13 -07:00
yulai.li
2347c21cdd Tune filer UI add rename feature 2022-04-07 15:19:41 +08:00
yulai.li
2454020a92 Add upload progress support 2022-04-07 15:19:37 +08:00
yulai.li
4f87ee7755 Add create directory and delete web UI features for filer 2022-04-07 15:19:33 +08:00
shibinbin
c20e1edd99 fix: master lose some volumes 2022-04-07 15:18:28 +08:00
eddy-gfx
ec53eec94f
Merge branch 'master' into a 2022-04-06 18:45:55 -05:00
chrislu
3ab2c0e5c0 avoid possible nil attributes 2022-04-06 10:11:11 -07:00
chrislu
9d44b89c21 Revert "avoid possible nil attributes" 
This reverts commit 0a6703c7f7.
 2022-04-06 10:01:45 -07:00
chrislu
0a6703c7f7 avoid possible nil attributes 2022-04-06 09:58:48 -07:00
Konstantin Lebedev
14a2cc83bf raft update peers via OnPeerUpdate 2022-04-06 21:17:04 +05:00
Konstantin Lebedev
357aa818fe add raft shell cmds 2022-04-06 15:23:53 +05:00
chrislu
3d229fe45c Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-04-06 00:28:51 -07:00
chrislu
79b8e6a8c3 add a place holder for later merge consecutive chunks 2022-04-06 00:28:48 -07:00
Konstantin Lebedev
c1aeded2dd force raft bootstrap 
avoid err bootstrap only works on new clusters
 2022-04-06 12:18:43 +05:00
Chris Lu
49ecb8d11c
Merge pull request #2879 from leyou240/master 
remove Redundant type conversion and use strings.TrimSuffix to enhance readability
 2022-04-06 00:02:19 -07:00
justin
d02f13c2d1 remove Redundant type conversion and use strings.TrimSuffix to enhance readability 2022-04-06 14:58:09 +08:00
root
f15a737a0a Add "Location:" in response when creating bucket 
according to "https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html"
 2022-04-06 13:56:27 +08:00
chrislu
bc888226fc erasure coding: tracking encoded/decoded volumes 
If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.
 2022-04-05 19:03:02 -07:00
chrislu
8b3d76b24d prevent concurrent access during shutdown 2022-04-05 10:53:36 -07:00
chrislu
bb4beebce3 prevent nil 2022-04-05 10:49:17 -07:00
Konstantin Lebedev
0e796a5582 rm set NoSnapshotRestoreOnStart 2022-04-05 18:28:42 +05:00
Konstantin Lebedev
68f11b9687 sleep bootstraping 2022-04-05 15:42:19 +05:00
Konstantin Lebedev
1ae7f509f5 LocalID from raw string 2022-04-05 13:50:39 +05:00
Konstantin Lebedev
89d32a0670 fix glog Info 2022-04-05 13:43:34 +05:00
Konstantin Lebedev
17c6e8e39f Merge branch 'new_master' into hashicorp_raft 
# Conflicts:
#	go.mod
#	go.sum
 2022-04-05 13:29:46 +05:00
Konstantin Lebedev
b7cdde14ae auto bootstraping and update peers 2022-04-05 13:17:53 +05:00
chrislu
4aae87f405 check missing hdd disk type 2022-04-04 14:48:00 -07:00
Konstantin Lebedev
622297f1a7 add stats raft handler 2022-04-04 19:16:06 +05:00
Konstantin Lebedev
1a1e5778c3 fix cluster status 2022-04-04 18:52:08 +05:00
Konstantin Lebedev
14dd971890 hashicorp raft with state machine 2022-04-04 17:51:51 +05:00
Konstantin Lebedev
c514710b7b initial add hashicorp raft 2022-04-04 13:50:56 +05:00
chrislu
d310711de0 avoid possible deadlock: move metadata log flush channel out of lock scope 2022-04-03 21:53:30 -07:00
chrislu
77a7d7253f 2.97 2022-04-03 19:08:01 -07:00
chrislu
e8d7bb42e2 avoid concurrent access to map object 
fix https://github.com/chrislusf/seaweedfs/issues/2866
 2022-04-03 01:08:25 -07:00
chrislu
6a2bcd03aa configure mount quota 2022-04-02 21:34:26 -07:00
chrislu
958f880b70 mount: add grpc method to adjust quota 2022-04-02 15:14:37 -07:00
chrislu
a8eefda452 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-04-01 20:25:37 -07:00
chrislu
105578a2f2 skip pinging self for master and volume server 2022-04-01 20:25:35 -07:00
root
b7e34016d4 fix error response in case of wrong expire time format 2022-04-02 11:01:33 +08:00
chrislu
4ecba915f3 add check between peers 2022-04-01 17:40:25 -07:00
chrislu
daca2d22a5 use original server address string as map key 2022-04-01 17:34:42 -07:00
chrislu
4b5c0e3fa9 check cluster connectivities 2022-04-01 17:27:49 -07:00
chrislu
6d55716fc3 use constants 2022-04-01 16:55:26 -07:00
chrislu
bbbbbd70a4 master supports grpc ping 2022-04-01 16:50:58 -07:00
chrislu
743ad690b6 filer supports grpc ping 2022-04-01 16:44:58 -07:00
chrislu
2305508b65 refactor: separate into two files 2022-04-01 16:40:49 -07:00
chrislu
800cbc004c volume server adds ping function 2022-04-01 16:37:06 -07:00
a
7e92517571 change user and pass to username and password 2022-04-01 14:09:25 -05:00
Konstantin Lebedev
7f1383a41e findExtraChunksInVolumeServers in consideration of replication 2022-04-01 14:45:41 +05:00
Konstantin Lebedev
3817e05dd0 fix collect filer files 2022-04-01 10:17:09 +05:00
Konstantin Lebedev
3cedb21bb7 skip new entities 2022-03-31 21:36:10 +05:00
Konstantin Lebedev
1d9e30d8c0 fsck replicas 2022-03-31 19:10:06 +05:00
Konstantin Lebedev
4f5b018a02 collecting only bucket volumes 2022-03-31 13:35:58 +05:00
Konstantin Lebedev
691ed50957 use constant for ErrBucketNotEmpty 2022-03-31 08:44:48 +05:00
Konstantin Lebedev
b6a2b43efe allowDeleteBucketNotEmpty 2022-03-30 22:46:13 +05:00
Konstantin Lebedev
a8b15c676c s3 test bucket delete nonempty 2022-03-30 21:06:26 +05:00
a
549e341c6f resolv 2022-03-30 06:31:19 -05:00
Chris Lu
ce8ef60c59
Merge pull request #2840 from kmlebedev/fix_multi_object_delete_key_limit 
fix test_s3.test_multi_object_delete_key_limit and test_multi_objectv2_delete_key_limit
 2022-03-30 01:27:13 -07:00
Konstantin Lebedev
6317f6a8a9 fix Description 2022-03-30 13:04:15 +05:00
Chris Lu
8732cc24c8
Merge pull request #2849 from guo-sj/update_user 
Add AWS IAM update user API and its test case
 2022-03-29 22:36:26 -07:00
Konstantin Lebedev
993283bb1d revert sort.Search 2022-03-29 19:53:12 +05:00
Konstantin Lebedev
5b90a39954 fix s3 tests: 
multipart_upload_incorrect_etag
multipart_resend_first_finishes_last
 2022-03-29 19:21:09 +05:00
Konstantin Lebedev
9231971c88 fix test_s3.test_multi_object_delete_key_limit and 
test_multi_objectv2_delete_key_limit
 2022-03-29 16:55:09 +05:00
guosj
ed07e76f0e add test function 2022-03-29 17:21:59 +08:00
a
bc603e534f diff 2022-03-28 16:50:28 +00:00
Konstantin Lebedev
84b7b83517 fix permission mkdir snapshot 
avoid open file operation not permitted
 2022-03-28 18:41:52 +05:00
Chris Lu
93615b2a49
Merge pull request #2828 from guo-sj/fix-tagging-error 
Fix delete all "Seaweed-" prefixed tagging error
 2022-03-28 00:35:22 -07:00
guosj
dd2b9d93cc update code according to PR's comment 2022-03-28 15:23:31 +08:00
guosj
02076f01a6 fix tagging error 2022-03-28 14:50:48 +08:00
justin
d51a724101 fix: encode small chunk return error maybe have some bug. 2022-03-28 13:11:24 +08:00
chrislu
0490ee87ef 2.96 2022-03-27 16:11:17 -07:00
chrislu
86d4c52d6e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-03-26 13:33:20 -07:00
chrislu
21e0898631 refactor: change masters from a slice to a map 2022-03-26 13:33:17 -07:00
chrislu
4ba7127ab1 refactor 2022-03-26 13:13:19 -07:00
Chris Lu
3153c836a7
Merge pull request #2744 from kmlebedev/issues_2545 
avoid connect to the old filler address
 2022-03-26 13:02:49 -07:00
chrislu
cae998eda1 Revert "Revert "remove duplicated metadata subscription in filer"" 
This reverts commit 34b743c481.
 2022-03-26 13:00:16 -07:00
chrislu
34b743c481 Revert "remove duplicated metadata subscription in filer" 
This reverts commit 34742be029.

Related to https://github.com/chrislusf/seaweedfs/issues/2545
 2022-03-26 12:33:45 -07:00
chrislu
fba1cfc2d6 simplify a bit 2022-03-26 10:24:05 -07:00
chrislu
a3411dd9da refactor 2022-03-26 10:21:26 -07:00
Chris Lu
1b27f96669
Merge pull request #2814 from kmlebedev/fix_remove_deleted_peers 
Fix remove deleted peers
 2022-03-25 11:36:15 -07:00
Konstantin Lebedev
ddd3945c26 fix remove deleted peers of raft server 
https://github.com/chrislusf/seaweedfs/issues/2804
 2022-03-25 15:09:38 +05:00
Konstantin Lebedev
c1450bf9fe always clear previous log to avoid server is promotable 
https://github.com/chrislusf/seaweedfs/issues/2804
 2022-03-25 13:40:19 +05:00
Chris Lu
89d84e275b
Merge pull request #2759 from kmlebedev/skip_wait_cancelled_request 
Need to exit waiting if request is was canceled
 2022-03-24 12:21:44 -07:00
Konstantin Lebedev
0b790d2bbf exclude the replication from the concurrentUploadLimitMB 2022-03-24 13:54:42 +05:00
chrislu
9b04f17555 remove dead code 2022-03-23 23:18:32 -07:00
chrislu
0b5faef612 fix 2022-03-23 23:16:54 -07:00
chrislu
0562fceb99 volume: fail fast if too many concurrent requests, to avoid dead lock due to replication. 
fix https://github.com/chrislusf/seaweedfs/issues/2755
 2022-03-23 22:53:58 -07:00
Konstantin Lebedev
849a2c1f68 fix cmdFiler 2022-03-23 14:30:46 +05:00
Konstantin Lebedev
105bcfdbd4 set iam ip from filer 2022-03-23 13:45:51 +05:00
chrislu
e48764be75 s3: multipart upload verifies uploaded parts 2022-03-23 01:05:14 -07:00
chrislu
18ae8943ef better clean up upload directory 2022-03-23 01:03:51 -07:00
chrislu
a0bad1c997 remove any go mod changes 
This reverts commit 6c7f7d6887.
 2022-03-21 23:04:00 -07:00
chrislu
6c7f7d6887 Revert "Merge pull request #2782 from SadmiB/upstream" 
This reverts commit a644b7236a, reversing
changes made to 349257f822.
 2022-03-21 23:00:50 -07:00
Chris Lu
0d692a6254
Merge branch 'master' into upstream 2022-03-21 11:02:04 -07:00
SadmiB
d12540c9f2 Add contabo api client 2022-03-21 17:16:49 +01:00
chrislu
4e2388e1b5 mount: set file size if it is only on remote gateway 2022-03-21 02:09:54 -07:00
chrislu
8f0410af2c 2.95 2022-03-21 01:47:03 -07:00
chrislu
ae558fa073 log reasons volumes became unwritable 2022-03-21 00:41:44 -07:00
chrislu
57c6eddd22 avoid possible deadlock if volume layout is used in some logs 2022-03-21 00:04:01 -07:00
chrislu
581c93d970 change to config_dir, to be consistent with "weed scaffold -config" 
fix https://github.com/chrislusf/seaweedfs/issues/2753
 2022-03-19 00:24:52 -07:00
chrislu
b75c6dcff5 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-03-19 00:22:52 -07:00
chrislu
3da2b83b38 Added a "-conf_dir" option to customize *.toml configuration file directory. 
fix https://github.com/chrislusf/seaweedfs/issues/2753
 2022-03-19 00:22:47 -07:00
elee
beb406bbbb fix ls, onBucketDelete still not triggering 2022-03-18 22:21:57 -05:00
elee
411c0df3fe switch to multi collection, change readme 2022-03-18 21:51:16 -05:00
zzq09494
9f16df736e fix-filer: calculation error of the method skipCheckParentDirEntry 2022-03-19 09:15:42 +08:00
elee
1cea6c73d3 update readme 2022-03-18 00:34:19 -05:00
elee
25be96832a update readme with index info 2022-03-18 00:33:08 -05:00
elee
06f23aa675 put in delete folder children query 2022-03-18 00:29:48 -05:00
elee
0701feeb17 added more to readme 2022-03-18 00:26:11 -05:00
elee
171c27ddf1 zz 2022-03-17 22:35:30 -05:00
elee
2f0cdcdceb update readme 2022-03-17 21:26:16 -05:00
elee
423ce57cde prefix search, bucket implemented 2022-03-17 21:12:25 -05:00
chrislu
2824940ecf Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-03-17 16:54:30 -07:00
chrislu
f247cab5cd skip localhost if bound to all interfaces already 0.0.0.0 or 127.0.0.1 2022-03-17 16:54:29 -07:00
Berck Nash
7ee38fa3a4 The fixes for https://github.com/chrislusf/seaweedfs/issues/1937 had a few problems: 
(1) The help file says that in the absence of a ipBind being specified, that it will bind to the "ip" specified.  Instead, it bound to localhost which broke the default configuration.  This change implements the documented behavior instead.

(2) The new IAM filer ip address has no default. This instantiates it to the same as the filer IP.  I'm not sure if there should be a corresponding iam.ip or iam.ipBind option added to the filer command?
 2022-03-17 15:30:23 -06:00
elee
bf745bdccb revise comment 2022-03-17 14:09:34 -05:00
elee
a0df993cef cleanup, add a few comments 2022-03-17 14:08:36 -05:00
elee
94883e2fad ok then 2022-03-17 05:23:12 -05:00
elee
f3ab6769e9 make start from indexing work 2022-03-17 05:09:13 -05:00
elee
31571dd96e fix file listing 2022-03-17 05:07:40 -05:00
elee
b25d03340b Merge branch 'a' of github.com:gfxlabs/seaweedfs into a 2022-03-17 04:51:59 -05:00
elee
921535001a arangodb adapter 2022-03-17 04:49:26 -05:00
chrislu
6886167d5f minor optimization 2022-03-17 00:03:28 -07:00
chrislu
7542fd7f4d mount: optimize a bit when writing files 2022-03-17 00:02:38 -07:00
chrislu
4042fdf3bb rename to skipCheckParentDir 
related to https://github.com/chrislusf/seaweedfs/pull/2761

It's better to default to false.
 2022-03-16 23:55:31 -07:00
zzq09494
40b0033fa7 go fmt 2022-03-17 14:19:48 +08:00
zzq09494
ee5417a5b3 fix: ut test error 2022-03-17 11:16:46 +08:00
zzq09494
81cce4b4c3 filer: support uploading file without needEnsureParentDir 2022-03-17 10:53:47 +08:00
zzq09494
a6a8892255 Revert "filer: support uploading file without needEnsureParentDir" 
This reverts commit a93c4947ba.
 2022-03-17 10:27:17 +08:00
zzq09494
a93c4947ba filer: support uploading file without needEnsureParentDir 2022-03-17 10:18:23 +08:00
Berck Nash
9b14f0c81a Add mTLS support for both master and volume http server. 2022-03-16 09:52:17 -06:00
chrislu
ed818c90a9 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-03-15 22:28:21 -07:00
chrislu
3639cad69c master, filer, s3: also listen to "localhost" in addition to specific ip address 
related to https://github.com/chrislusf/seaweedfs/issues/1937
 2022-03-15 22:28:18 -07:00
Konstantin Lebedev
f43c6daeda Need to exit waiting if request is was canceled 2022-03-15 19:55:22 +05:00
Konstantin Lebedev
c2e7d663bc reduced the timeout for creating a connection from 30 to 10 seconds 2022-03-15 13:32:05 +05:00
chrislu
fbc9f0eb64 minor 2022-03-14 03:19:16 -07:00
chrislu
2eda3a686f 2.94 2022-03-14 00:55:01 -07:00
chrislu
bd5c5586b5 generate inode via path and time 2022-03-14 00:03:29 -07:00
chrislu
5cba8e51c5 refactor 2022-03-13 18:34:57 -07:00
chrislu
f2f68f675e write to disk during random writes, limiting total disk spaces used 2022-03-13 18:17:35 -07:00
chrislu
2f4cd20f90 tests skip reader pattern monitoring 2022-03-13 18:15:53 -07:00
chrislu
53513475bf mount: add back random read support 
avoid too much memory used also
 2022-03-13 01:38:52 -08:00
chrislu
f70c1e449b add useful doc link 2022-03-13 00:14:50 -08:00
chrislu
b20ddc57a7 mount: return open status 2022-03-12 22:38:14 -08:00
chrislu
b40d252761 mount: chmod for root 2022-03-12 12:10:56 -08:00
chrislu
3a6eb8ca5f default bind to one ip address 
fix https://github.com/chrislusf/seaweedfs/issues/1937
 2022-03-11 14:02:39 -08:00
Konstantin Lebedev
f53cff045f checks disk file exist 2022-03-10 18:58:56 +05:00
Konstantin Lebedev
834210a9dc avoid connect to the old filler address 
https://github.com/chrislusf/seaweedfs/issues/2545
 2022-03-10 15:24:45 +05:00
chrislu
197ade6aeb Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-03-09 22:26:55 -08:00
chrislu
011a41b561 add back writes to swap file when too many in memory chunks are used. 2022-03-09 22:26:51 -08:00
zzq09494
2ea18fca48 fix:support some databases which not support 'IF NOT EXISTS' keyword of 'create table command' 2022-03-10 13:37:34 +08:00
banjiaojuhao
f28dbbe5c5 [bugfix] filer: 1. Delete uploaded chunks when upload failed. 2. Report error when upload is interrupted by user. 2022-03-10 11:40:39 +08:00
chrislu
e2b07737da Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-03-08 23:02:33 -08:00
chrislu
dc204dd137 fix nil entry 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1d340b4]

goroutine 130523 [running]:
github.com/chrislusf/seaweedfs/weed/filer.FileSize(...)
        /code/seaweedfs/weed/filer/filechunks.go:26
github.com/chrislusf/seaweedfs/weed/mount.(*WFS).Lookup(0xc000866d80, 0x1, 0xc002897f40, {0xc004b00980, 0x39}, 0x1ec19e0)
        /code/seaweedfs/weed/mount/weedfs_dir_lookup.go:59 +0x654
github.com/hanwen/go-fuse/v2/fuse.doLookup(0xc00033c000, 0xc00033c000)
        /code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/opcode.go:333 +0x6b
github.com/hanwen/go-fuse/v2/fuse.(*Server).handleRequest(0xc000ab2420, 0xc00033c000)
        /code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/server.go:483 +0x1f3
github.com/hanwen/go-fuse/v2/fuse.(*Server).loop(0xc000ab2420, 0x0)
        /code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/server.go:456 +0x110
created by github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest
        /code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/server.go:323 +0x534
 2022-03-08 23:02:30 -08:00
banjiaojuhao
f7f2a597dd minor 2022-03-08 16:22:55 +08:00
banjiaojuhao
d61bea9038 [bugfix] filer: In file modification, old chunks will be mis-deleted when they are merged(Manifestized). 2022-03-08 16:22:55 +08:00
chrislu
3aeee3d748 ensure releasing file handle 2022-03-07 14:01:24 -08:00
chrislu
8136384473 remove debug message 2022-03-07 11:22:26 -08:00
banjiaojuhao
b9ff7723dd [bugfix] filer: nil pointer dereference 2022-03-07 23:26:25 +08:00
chrislu
da3d330616 s3 and filer transport using unix domain socket instead of tcp 2022-03-07 02:00:14 -08:00
chrislu
0cb17b45b1 refactoring 2022-03-07 01:59:01 -08:00
chrislu
6d3db4445b buffer for all range requests 2022-03-07 01:56:47 -08:00
chrislu
f3bcbeb60a a little optimization 2022-03-07 00:24:59 -08:00
chrislu
f7ee60996c Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-03-07 00:08:09 -08:00
chrislu
6f3ec989a7 fix manifest batch size 2022-03-07 00:07:53 -08:00
Chris Lu
1e7fcef581
Merge pull request #2729 from banjiaojuhao/filer_metadata-resolve-manifest 
filer: support get metadata with resolved manifest chunk
 2022-03-07 00:05:46 -08:00
chrislu
bb0b784544 minor 2022-03-07 00:04:59 -08:00
banjiaojuhao
bfcc9ca808 filer: support metadata with resolved manifest chunk 2022-03-07 15:47:51 +08:00
banjiaojuhao
71f3046841 filer: add back isAppend function 2022-03-07 15:41:07 +08:00
chrislu
0ba4e4cd23 2.93 2022-03-06 18:54:12 -08:00
chrislu
46a28b8819 mount: adjust disk space based on quota 2022-03-06 17:22:49 -08:00
chrislu
ede6ce44c6 fix test 2022-03-06 17:09:55 -08:00
chrislu
21ef152423 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-03-06 17:04:24 -08:00
chrislu
f3442e36e6 mount: quota adjust error type to be syscall.ENOSPC 2022-03-06 17:04:21 -08:00
Chris Lu
89cd0c27a8
Merge pull request #2728 from kmlebedev/set_default_leveldb2 2022-03-06 07:26:30 -08:00
Konstantin Lebedev
cf444ebd07 Set default leveldb2 enabled 
avoid Filer store is enabled for both leveldb2 and mysql
 2022-03-06 18:27:25 +05:00
banjiaojuhao
a07c93f4dd filer: reset bytesBuffer before use 2022-03-06 21:07:36 +08:00
chrislu
c7e8ac18f0 mount: quota for one mounted collection 
related to https://github.com/seaweedfs/seaweedfs-csi-driver/issues/48
 2022-03-06 02:44:40 -08:00
chrislu
b7c992f410 add flag to enforce quota 2022-03-05 22:10:43 -08:00
chrislu
f1713c96ae avoid possible runtime error: index out of range [0] with length 0 2022-03-05 21:14:31 -08:00
chrislu
da76af187f mount: avoid possible index out of bounds error 2022-03-04 22:36:01 -08:00
chrislu
6e49e75a5b use logical number of files and sizes for statistics and quota 2022-03-04 18:47:44 -08:00
chrislu
f51e20028a mount: avoid comma in mount options 
fix https://github.com/chrislusf/seaweedfs/issues/2719
 2022-03-03 03:42:29 -08:00
chrislu
dc0f48682c unused 2022-03-03 03:41:35 -08:00
chrislu
28b8974a3a mount: fix directory pagination when using midnight commander 2022-03-03 02:59:31 -08:00
chrislu
a96d4254e9 filer, s3, volume server: a bit memory optimization 2022-03-02 20:15:28 -08:00
chrislu
6fbbc78574 stream reading a whole chunk 2022-03-02 13:50:46 -08:00
chrislu
784583afc6 avoid pool memory allocation if too large 2022-03-02 13:50:28 -08:00
chrislu
ba14307319 2.92 2022-02-28 15:22:19 -08:00
chrislu
fcf3714443 mount: add back support for filer.path 2022-02-28 12:16:53 -08:00
chrislu
80c017907b filer.backup: fix backing up encrypted chunks 
I have done filer.backup test:
replication.toml:
[sink.local]
enabled = true
directory = "/srv/test"
___
system@dat1:/srv/test$ weed filer.backup -filer=app1:8888 -filerProxy
I0228 12:39:28 19571 filer_replication.go:129] Configure sink to local
I0228 12:39:28 19571 filer_backup.go:98] resuming from 2022-02-28 12:04:20.210984693 +0100 CET
I0228 12:39:29 19571 filer_backup.go:113] backup app1:8888 progressed to 2022-02-28 12:04:20.211726749 +0100 CET 0.33/sec

system@dat1:/srv/test$ ls -l
total 16
drwxr-xr-x 2 system system 4096 Feb 28 12:39 a
-rw-r--r-- 1 system system   48 Feb 28 12:39 fu.txt
-rw-r--r-- 1 system system   32 Feb 28 12:39 _index.html
-rw-r--r-- 1 system system   68 Feb 28 12:39 index.php
system@dat1:/srv/test$ cat fu.txt
?	?=?^??`?f^};?{4?Z%?X0=??rV????|"?1??踪~??
system@dat1:/srv/test$
On the active mount on the target server it's:
system@app1:/srv/app$ ls -l
total 2
drwxrwxr-x 1 system system  0 Feb 28 12:04 a
-rw-r--r-- 1 system system 20 Feb 28 12:04 fu.txt
-rw-r--r-- 1 system system  4 Feb 28 12:04 _index.html
-rw-r--r-- 1 system system 40 Feb 28 12:04 index.php
system@app1:/srv/app$ cat fu.txt
This is static boy!
Filer was started with: weed filer master="app1:9333,app2:9333,app3:9333" -encryptVolumeData
It seems like it's still encrypted?
 2022-02-28 10:07:06 -08:00
chrislu
aad62ee148 mount: mark directory uncached if forgotten 2022-02-28 02:08:24 -08:00
chrislu
554e239097 release file handle 2022-02-28 01:23:14 -08:00
chrislu
3639fedd01 mount: fix fsync opened and renamed files 2022-02-28 00:34:17 -08:00
chrislu
63a9d8f01d ensure inodes are not duplicating unless hardlinked 2022-02-27 23:13:49 -08:00
chrislu
de77d00c81 correctly clean up for a file 2022-02-27 23:12:28 -08:00
chrislu
18543c6e8b minor 2022-02-27 23:11:09 -08:00
chrislu
09cd00f356 2.91 2022-02-27 04:03:39 -08:00
chrislu
941ced60a4 download 2 chunks if at the beginning of a file 2022-02-27 03:57:24 -08:00
chrislu
f9d9eed0c9 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-27 03:41:35 -08:00
chrislu
d602d68fd1 remove dead code 2022-02-27 03:41:32 -08:00
Chris Lu
6c07af6014
Merge pull request #2715 from gfx-labs/acl-pr 2022-02-27 03:27:32 -08:00
chrislu
c3792c8352 remove dead code 2022-02-27 03:03:19 -08:00
chrislu
aa9eef81e6 retire mount v1 2022-02-27 02:57:27 -08:00
eddy-gfx
fb940dd807
Merge branch 'chrislusf:master' into acl-pr 2022-02-27 04:52:08 -06:00
elee
881a0fe806 ensure compatibility 2022-02-27 04:50:59 -06:00
elee
954ad98e0d set canned acl on replication create 2022-02-27 04:49:31 -06:00
chrislu
6e1ab97988 use debug option to see operations 2022-02-27 02:02:30 -08:00
chrislu
2112d99140 mount2: add back readonly mode 2022-02-27 01:13:32 -08:00
chrislu
9ef5bb20f6 mount2: invalidate fuse cache for replaced inode 2022-02-27 00:00:23 -08:00
chrislu
4ddcbaab57 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-26 23:20:47 -08:00
chrislu
551d00d51a prefetch other chunks when stream reading 2022-02-26 23:20:45 -08:00
Chris Lu
e1d3dd5e18
Merge pull request #2712 from guo-sj/correct_comments 2022-02-26 09:16:12 -08:00
guosj
82cad5e330
correct comments 2022-02-26 22:44:26 +08:00
chrislu
7b1a713d2a remove dead code 2022-02-26 03:23:15 -08:00
chrislu
3a58b7bac6 a little safer 2022-02-26 03:23:06 -08:00
chrislu
708e14fcfa avoid possible too big memory allocation 2022-02-26 03:22:41 -08:00
chrislu
3345a50d9b prefetch 2 chunks 2022-02-26 03:06:17 -08:00
chrislu
86ce69f709 remove logs 2022-02-26 03:00:20 -08:00
chrislu
b2a148cb4c use file size as max range 2022-02-26 03:00:08 -08:00
chrislu
2ab0ad24a3 use memory pool 2022-02-26 02:59:19 -08:00
chrislu
28b395bef4 better control for reader caching 2022-02-26 02:16:47 -08:00
chrislu
3ad5fa6f6f chunk cache adds function ReadChunkAt 2022-02-25 21:55:04 -08:00
chrislu
fc7a4957ea fix mount2 options 2022-02-25 21:22:44 -08:00
chrislu
72c0233938 less logs 2022-02-25 15:34:24 -08:00
chrislu
101e6d80b0 mount2: listXattr return ok if xattr is empty 2022-02-25 14:38:56 -08:00
chrislu
03466f955e rename: delete source entry metadata only, skipping hard links 2022-02-25 02:57:54 -08:00
chrislu
8080fe4cc1 logs 2022-02-25 02:56:23 -08:00
chrislu
e423548673 rename: pass along entry metadata 2022-02-25 02:53:37 -08:00
chrislu
e8110bb54c Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-25 01:17:28 -08:00
chrislu
202a29d014 refactoring 2022-02-25 01:17:26 -08:00
chrislu
91d6785cf3 define metadata action types 2022-02-25 00:54:16 -08:00
chrislu
be3fc77391 mount2: use consistent inode 2022-02-25 00:53:27 -08:00
Chris Lu
9873bae115
Merge pull request #2706 from guo-sj/fix_log_info 2022-02-24 22:48:19 -08:00
guosj
121b31f750 fix incorrect log information 2022-02-25 13:41:20 +08:00
chrislu
ceaf993a27 mount2: add rdev 2022-02-24 14:51:25 -08:00
chrislu
95717d1006 simpler output 2022-02-24 13:50:08 -08:00
chrislu
b05962b90e rename: handle hard links 2022-02-24 02:59:00 -08:00
chrislu
e31ec04f4f mount2: POSIX deleted opened file nlink should be 0 2022-02-24 01:59:37 -08:00
chrislu
419e355e9e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-24 01:31:07 -08:00
chrislu
91f0481f4e mount2: SetAttr set mode correctly 2022-02-24 01:31:04 -08:00
Chris Lu
97a4b66df7
Merge pull request #2704 from guo-sj/fix_bugs_in_return_value 
fix return value in storage/volume_vacuum.go:444
 2022-02-24 00:49:29 -08:00
chrislu
abe7214c1f Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-24 00:37:59 -08:00
chrislu
b93d57da31 mount2: dir read opened file 2022-02-24 00:37:57 -08:00
guosj
3e7aa1caf5 fix return value in storage/volume_vacuum.go:444 2022-02-24 15:54:36 +08:00
Chris Lu
6834df77a0
Merge pull request #2701 from guo-sj/fix_bugs_in_return_value 2022-02-23 19:23:37 -08:00
guosj
26f3ab8d4b fix a return bug in func (c *commandVacuum) Do 2022-02-24 09:04:38 +08:00
chrislu
320637dc7a use "mv.from" for moving files 2022-02-23 15:34:42 -08:00
chrislu
c29bc9a367 fix error handling 2022-02-23 15:34:25 -08:00
Chris Lu
da58c748bc
Merge pull request #2698 from guo-sj/fix_bugs_in_return_value 2022-02-23 00:41:30 -08:00
Chris Lu
bd092d8318
Merge pull request #2697 from guo-sj/fix_bugs_in_return_value 2022-02-23 00:30:52 -08:00
guosj
d68c27f82d fix another return value bug 2022-02-23 16:21:25 +08:00
guosj
8f9aa0cddd fix bugs in return value 2022-02-23 16:17:48 +08:00
Tuan Vuong
d2ec62656d initialize master address in iam options 2022-02-23 12:01:54 +07:00
banjiaojuhao
6ab09e9071 filer_http: support uploading file with offset 2022-02-22 00:15:00 +08:00
chrislu
497ebbbd45 2.90 2022-02-20 22:00:13 -08:00
chrislu
56da3494cb leveldb3: adjust memory allocation for each bucket 2022-02-20 13:40:13 -08:00
banjiaojuhao
e6126cef62 filer_web: support moving entry 2022-02-20 23:56:23 +08:00
chrislu
7bc67399e4 listing for filer stores without prefixed query: break if no more progress 2022-02-19 10:12:51 -08:00
chrislu
248c0c8087 minor 2022-02-19 09:54:09 -08:00
chrislu
4ee0a6f47b filer store: reduce one possible listing operation 2022-02-19 00:43:42 -08:00
chrislu
b3594278c9 optimize a bit 2022-02-18 23:36:10 -08:00
chrislu
61811dc2f1 comments 2022-02-18 22:14:40 -08:00
chrislu
daa27b2119 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-18 20:55:10 -08:00
chrislu
9014d00fd0 Revert "s3: listObjectParts return ErrNoSuchUpload if does not exist" 
This reverts commit 6cf2e7d493.
 2022-02-18 20:54:54 -08:00
banjiaojuhao
4c30934cd9 filer: support get file entry 2022-02-18 22:52:26 +08:00
chrislu
63062ed7f0 mount2: fix unlink 2022-02-18 01:10:53 -08:00
chrislu
93e12d5f30 fix build 2022-02-18 00:48:00 -08:00
chrislu
d62370d4e0 add todo 2022-02-18 00:47:15 -08:00
chrislu
e8ce30fdc5 mount2: adjust file mode 2022-02-18 00:47:02 -08:00
chrislu
f9d33f70b0 return fuse.Status when looking up by inode 2022-02-18 00:45:43 -08:00
chrislu
b9cf4f12fc Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-16 21:33:51 -08:00
chrislu
49b84b6e2a list entries while reading from remote 2022-02-16 21:32:15 -08:00
Chris Lu
38dcaaa76e
Merge pull request #2683 from guo-sj/fix_fsconfigure_bug 
fix minor bug in commandFsConfigure.Do
 2022-02-16 18:16:47 -08:00
guosj
ca121ecd66 fix minor bug in commandFsConfigure.Do 2022-02-17 09:57:52 +08:00
chrislu
65a19e3abc fix listing with correct inode 2022-02-16 17:01:39 -08:00
chrislu
6ac066d1dc count lookup or not 2022-02-16 16:49:03 -08:00
chrislu
a129bda7d9 sync data first before stopping 2022-02-16 09:11:34 -08:00
chrislu
118d0e01a8 less logs 2022-02-16 08:45:07 -08:00
chrislu
a6bc67c34c less logs 2022-02-16 08:38:51 -08:00
chrislu
2facd65998 fix second listing 2022-02-16 08:16:27 -08:00
chrislu
1013fc90d8 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-16 07:57:11 -08:00
chrislu
22739c653a clean up 2022-02-16 07:57:08 -08:00
Chris Lu
3cb19cf6db
Merge pull request #2680 from kmlebedev/volume_healthz 2022-02-16 04:43:44 -08:00
Konstantin Lebedev
9ea09cc41c healthz check to avoid drain pod with last replicas 2022-02-16 14:18:36 +05:00
chrislu
65bfeafb5a add back mkdir 2022-02-16 01:09:21 -08:00
chrislu
37e8fce841 clean up cache on exit 2022-02-16 00:39:21 -08:00
chrislu
1560ec7e26 remove unused code 2022-02-16 00:37:24 -08:00
chrislu
3cbce878f2 mount2: fix directory pagination 2022-02-15 22:42:10 -08:00
chrislu
df51e0c042 mongodb: remove ErrNoDocuments checking since it is only for creating single record, not for querying 2022-02-15 16:18:36 -08:00
chrislu
98cce6a150 mongodb: ensure closing cursor 2022-02-15 16:15:44 -08:00
Chris Lu
aa7fc299b8
Merge pull request #2676 from banjiaojuhao/add_filer_store-redis_lua 
FilerStore: add redis_lua
 2022-02-15 12:10:34 -08:00
banjiaojuhao
fc3b75f2f8 weed/operation/delete_content.go: nil pointer dereference 2022-02-15 23:07:03 +08:00
banjiaojuhao
b5ec346700 FilerStore: add redis_lua 2022-02-15 20:54:57 +08:00
chrislu
17ac5244c3 mount2: avoid double listing directories 2022-02-15 01:44:17 -08:00
guosj
db7766323e complete project code, remain test code 2022-02-15 17:35:33 +08:00
chrislu
222798d926 mount2: fix for read dir plus on linux 2022-02-15 00:29:17 -08:00
chrislu
4e72863ba5 mount2 add debug mode 2022-02-15 00:26:30 -08:00
Chris Lu
64afbc5235
Merge pull request #2640 from Radtoo/fsck_with_delete 
Added basic deletion capabilities to fsck.
 2022-02-14 23:00:01 -08:00
chrislu
4e181db21a mount: default disable cache 
* Prevent cases as https://github.com/seaweedfs/seaweedfs-csi-driver/issues/43
* Improve read write benchmarks
* Improve AI training performance. Most of the files are just read once.
 2022-02-14 20:42:33 -08:00
chrislu
ebc22625b4 less noisy on mac 2022-02-14 16:38:01 -08:00
chrislu
377bf31445 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-14 13:48:51 -08:00
chrislu
e8420aaed7 fix building for windows freebsd 2022-02-14 13:48:48 -08:00
Chris Lu
d3ee621fce
Merge pull request #2661 from garenchan/ck-dev1 2022-02-14 10:08:37 -08:00
Konstantin Lebedev
526094d2da StopTimeout 30 sec 2022-02-14 21:42:27 +05:00
Konstantin Lebedev
275e9a4e86 reduce to default http server KillTimeout and StopTimeout 2022-02-14 21:38:24 +05:00
garenchan
bd032eabe7 [UPDATE] Make heartbeat interval and election timeout of masters configurable. 2022-02-14 21:09:07 +08:00
chrislu
ff666104c4 fix GOOS 2022-02-14 03:14:05 -08:00
chrislu
05724a68d4 skip other OS 2022-02-14 02:59:51 -08:00
chrislu
4244ef8b72 fix building on linux 2022-02-14 02:59:27 -08:00
chrislu
b9c2bff931 clean up 2022-02-14 02:14:26 -08:00
chrislu
a990cd29cd Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-14 01:52:25 -08:00
chrislu
6a40fd1c65 2.89 2022-02-14 01:52:16 -08:00
Chris Lu
305418fac5
Merge pull request #2667 from kmlebedev/syncReplicatedWrite 
atomically write to replicas
 2022-02-14 01:48:21 -08:00
Chris Lu
aa13168b4d
Merge pull request #2668 from chrislusf/mount2 
Mount2
 2022-02-14 01:41:09 -08:00
chrislu
fe57a2e770 file set attribute 2022-02-14 01:36:10 -08:00
Konstantin Lebedev
0ed76a0556 clearly 2022-02-14 14:10:06 +05:00
chrislu
dbeeda8123 listen for metadata updates 2022-02-14 01:09:31 -08:00
Konstantin Lebedev
36013f63ed https://github.com/chrislusf/seaweedfs/issues/2648 2022-02-14 13:59:12 +05:00
chrislu
7286e525ad support write 2022-02-13 23:27:11 -08:00
chrislu
2b955c1713 support read 2022-02-13 22:50:44 -08:00
chrislu
f3c1e00521 rename 2022-02-13 19:16:56 -08:00
chrislu
bb9919b07a add open release, refactor 2022-02-13 19:14:34 -08:00
Eng Zer Jun
b92df1654c
test: use T.TempDir to create temporary test directory 
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2022-02-14 10:38:13 +08:00
chrislu
072f923a9b add doc 2022-02-13 16:56:35 -08:00
chrislu
3d0e9e5197 supports renaming 2022-02-13 16:34:57 -08:00
chrislu
417b59b893 doc 2022-02-13 14:37:47 -08:00
chrislu
24290fed9d add comments 2022-02-13 06:02:21 -08:00
chrislu
6200b6abb1 avoid fatal message in some edge cases 2022-02-13 05:59:10 -08:00
chrislu
6a921e15f3 forget() factor in nlookup 2022-02-13 05:49:29 -08:00
chrislu
f8af0f93d9 support link 2022-02-13 05:29:43 -08:00
chrislu
0381338d40 fix import 2022-02-13 04:23:06 -08:00
chrislu
a1ef0e48a9 doc 2022-02-13 04:22:02 -08:00
chrislu
be40ff6048 add symlink 2022-02-13 03:50:16 -08:00
chrislu
3d93570979 supports forget 2022-02-13 03:31:47 -08:00
chrislu
6a42cb6b0b supports mknod, unlink 2022-02-13 03:09:24 -08:00
chrislu
813b868b9a add rmdir 2022-02-13 01:43:11 -08:00
chrislu
e85ca10a1a add mkdir 2022-02-13 01:34:19 -08:00
chrislu
21046c6a28 split files 2022-02-13 01:05:30 -08:00
chrislu
a4c9223b9d support xattr 2022-02-13 00:58:46 -08:00
chrislu
c81833a192 add directory setAttr 2022-02-12 23:08:56 -08:00
chrislu
5c48c23235 remove println 2022-02-12 22:45:07 -08:00
chrislu
7cfbf1e85f fix ok status 2022-02-12 22:41:45 -08:00
chrislu
4c75fd5f9c sync format 2022-02-12 22:41:29 -08:00
chrislu
b0a5193e32 working 2022-02-12 22:21:30 -08:00
chrislu
661a34e23d Merge branch 'master' into mount2 2022-02-12 17:58:32 -08:00
chrislu
5e814afe88 blocks count 2022-02-12 17:58:12 -08:00
Radtoo
724ce04b1c We picked the second flag's name. 2022-02-13 00:53:35 +01:00
chrislu
5a0a709016 it runs, but directory listing output is not showing up 2022-02-12 05:27:16 -08:00
chrislu
866981d8ac rename 2022-02-12 02:49:15 -08:00
chrislu
72faae91e1 implement read directory and read directory plus 2022-02-12 02:48:44 -08:00
chrislu
a10c28ba82 simplify 2022-02-12 01:59:36 -08:00
chrislu
f4d88862c4 can attr root directory 2022-02-12 01:54:16 -08:00
chrislu
180445f5a8 change to use fuse file system 2022-02-11 21:35:09 -08:00
chrislu
45a0fda9bd need to follow https://github.com/hanwen/go-fuse/blob/master/fuse/api.go 2022-02-11 03:12:52 -08:00
chrislu
f87da798a4 to be re-written following fuse virtual file system 2022-02-11 03:09:30 -08:00
chrislu
9a913457dd supports stats 2022-02-10 23:23:47 -08:00
chrislu
b6143de52a mount with name 2022-02-10 22:43:55 -08:00
chrislu
7a0c35674c clean up previously mounted folder 2022-02-10 20:46:53 -08:00
chrislu
c3f9d9fa2e initial setup 2022-02-10 20:32:13 -08:00
Radtoo
fbe7ed7927 Revert the the flag to "reallyDeleteFromVolume". 2022-02-10 20:01:48 +01:00
chrislu
21aaa4c1f1 ec.encode: calculate free ec slots based on (maxVolumeCount-volumeCount) 
fix https://github.com/chrislusf/seaweedfs/issues/2642
 2022-02-08 01:51:13 -08:00
chrislu
13f6ec1c4e test checking ec distribution 2022-02-08 01:50:05 -08:00
chrislu
f34c2ff7c5 use embed txt 2022-02-08 01:14:27 -08:00
chrislu
c8c7c10c3f volume.tier.move: avoid double counting 
related to https://github.com/chrislusf/seaweedfs/issues/2637
 2022-02-08 00:57:35 -08:00
chrislu
f18803424a volume.balance: add delay during tight loop 
fix https://github.com/chrislusf/seaweedfs/issues/2637
 2022-02-08 00:53:55 -08:00
chrislu
9860405974 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-07 23:08:57 -08:00
chrislu
85c1615b43 filer read empty file may cause OOM in some cases 
fix https://github.com/chrislusf/seaweedfs/issues/2641
 2022-02-07 23:08:54 -08:00
Chris Lu
a2ac540ecc
Merge pull request #2645 from guol-fnst/fix_Precedence 
fix preconditions
 2022-02-07 18:32:16 -08:00
root
7f0c793083 fix preconditions according to https://tools.ietf.org/id/draft-ietf-httpbis-p4-conditional-26.html#preconditions 2022-02-08 10:13:19 +08:00
chrislu
b1cff07ab0 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-07 11:38:01 -08:00
chrislu
81f86c381d s3: avoid nil response 
fix https://github.com/chrislusf/seaweedfs/issues/2636
 2022-02-07 11:37:59 -08:00
Chris Lu
f46763d74f
Merge pull request #2644 from kmlebedev/vacum_fsync 
updated needle with fsync
 2022-02-07 11:33:58 -08:00
Konstantin Lebedev
ef541972f8 updated needle with fsync 2022-02-08 00:10:53 +05:00
Chris Lu
1b7fb3ce84
Merge pull request #2643 from kmlebedev/graceful_volume_stop 2022-02-07 09:58:26 -08:00
Konstantin Lebedev
fb97e234c9 skips compact if store is stopping 2022-02-07 20:16:15 +05:00
chrislu
9405eaefdb filer.sync: fix replicating partially updated file 
Run two servers with volumes and fillers:
server -dir=Server1alpha -master.port=11000 -filer -filer.port=11001 -volume.port=11002
server -dir=Server1sigma -master.port=11006 -filer -filer.port=11007 -volume.port=11008

Run Active-Passive filler.sync:
filer.sync -a localhost:11007 -b localhost:11001 -isActivePassive

Upload file to 11007 port:
curl -F file=@/Desktop/9.xml "http://localhost:11007/testFacebook/"

If we request a file on two servers now, everything will be correct, even if we add data to the file and upload it again:
curl "http://localhost:11007/testFacebook/9.xml"
EQUALS
curl "http://localhost:11001/testFacebook/9.xml"

However, if we change the already existing data in the file (for example, we change the first line in the file, reducing its length), then this file on the second server will not be valid and will not be equivalent to the first file

Снимок экрана 2022-02-07 в 14 21 11

This problem occurs on line 202 in the filer_sink.go file. In particular, this is due to incorrect mapping of chunk names in the DoMinusChunks function. The names of deletedChunks do not match the chunks of existingEntry.Chunks, since the first chunks come from another server and have a different addressing (name) compared to the addressing on the server where the file is being overwritten.

Deleted chunks are not actually deleted on the server to which the file is replicated.
 2022-02-07 03:46:28 -08:00
Radtoo
fbb14e0ea8 Adding separate toggle to purge absent vols 
While this toggle is basically required to clean out entries for deleted volumes, having a separate description + toggling this separately seems like a good idea so people get a chance to check if their volumes are all mounted/connected as expected.

Also renamed forcePurge to just purge.
 2022-02-06 23:46:52 +01:00
Radtoo
6b17f45da2 Also delete paths for which a volume is entirely absent. 2022-02-06 23:22:04 +01:00
Radtoo
fa0cfdfb7f Added basic deletion capabilities to fsck. 2022-02-06 17:05:58 +01:00
chrislu
433fde4b18 move error to a separate file 
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
 2022-02-04 22:57:51 -08:00
Chris Lu
a23fcb9a7c
Merge pull request #2634 from kmlebedev/errorMetrics 
error metrics for filer and store
 2022-02-04 22:35:13 -08:00
Chris Lu
247bbabda5
Merge pull request #2632 from lapshin-vitaly/s3api_errors 
add s3api error for copy in file, not directory
 2022-02-04 22:34:34 -08:00
chrislu
ced3b89395 add util package 2022-02-04 21:34:58 -08:00
chrislu
affe3c2c12 change to util.WriteFile 2022-02-04 21:32:27 -08:00
chrislu
76e297d64f sync call to write file, avoid vif loading error 
fix https://github.com/chrislusf/seaweedfs/issues/2633
 2022-02-04 11:14:04 -08:00
zerospiel
f3364fec99 weed/s3api: rearrange s3 methods handlers to ensure correct methods requesting 
Otherwise current calls for some methods (i.e. GetObjectAcl) ends up with wrong method selection (i.e. GetObject).

Added generic comment rule of traversing methods
 2022-02-04 15:14:48 +03:00
Konstantin Lebedev
9978f54acf fix metric names 2022-02-04 16:45:16 +05:00
Konstantin Lebedev
3f4e17aa24 error metrics for filer and store 2022-02-04 14:07:14 +05:00
Lapshinn Vitaly
6bdc274d4d add s3api error for copy in file, not directory 2022-02-04 03:28:37 +03:00
chrislu
6cf2e7d493 s3: listObjectParts return ErrNoSuchUpload if does not exist 
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api abort-multipart-upload --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api list-parts --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
{
    "Initiator": null,
    "Owner": null,
    "StorageClass": "STANDARD"
}

If we abort a multipart upload, it appears that records are left behind. We should get a 404 NoSuchKey error.
 2022-02-03 12:34:16 -08:00
zerospiel
b54a65ba5a weed/s3api: added new bucket handlers for more compatibility with AWS S3 
Protocol

Otherwise any requests to the underlying handlers results in calls to
ListObjects (v1) that may intensively load gateway and volume servers.

Added the following handlers with default responses:
- GetBucketLocation
- GetBucketRequestPayment

Added the following handlers with NotFound and NotImplemented responses:
- PutBucketAcl
- GetBucketPolicy
- PutBucketPolicy
- DeleteBucketPolicy
- GetBucketCors
- PutBucketCors
- DeleteBucketCors
 2022-02-03 17:17:05 +03:00
chrislu
6bee1e9714 [volume.check.disk] fix wrong logic to compare 
fix https://github.com/chrislusf/seaweedfs/issues/2626

minuend - subtrahend
 2022-02-01 07:48:28 -08:00
chrislu
7270067289 2.88 2022-01-30 20:25:26 -08:00
chrislu
84c9bc4389 edge case: old entry was not replicated to remote storage 2022-01-30 20:23:24 -08:00
chrislu
b8490fe427 adjust volume count even when not applying the changes 2022-01-28 19:11:46 -08:00
Chris Lu
b3f0f170b4
Merge pull request #2618 from divanikus/master 2022-01-28 03:34:52 -08:00
chrislu
b9b684194f remove max connection age 
following https://github.com/grpc/grpc-go/issues/3170#issuecomment-552517779
 2022-01-27 02:28:22 -08:00
divanikus
67e3fe996a async volumeTierMove 2022-01-26 18:22:31 +03:00
chrislu
62d815d1ca use limited in memory buffer instead of swap file 2022-01-26 00:36:19 -08:00
Chris Lu
9596fce562
Merge pull request #2613 from kmlebedev/masterPromMetrics 
Master prom metrics
 2022-01-25 08:13:14 -08:00
Konstantin Lebedev
5c9259fa3c fix metrics master name 2022-01-25 14:42:47 +05:00
chrislu
7328962009 revert "POSIX: should not delete if a directory is not empty" 
revert 0c75f15062

Reported:

Hi, about commit: 0c75f15062
POSIX: should not delete if a directory is not empty
It should still delete with a command like rm -rf "${path}"/  because it is a forced delete, but now it gets fail to delete non-empty folder: [...]
Can you enable the delete if it is forced?
 2022-01-24 14:21:50 -08:00
Konstantin Lebedev
c9952759c4 metrics master is leader 2022-01-24 20:13:07 +05:00
Konstantin Lebedev
28efe31524 new master metrics 2022-01-24 19:09:43 +05:00
chrislu
3bba2124ef use a sliding window of in-memory writable chunks 2022-01-23 23:02:05 -08:00
chrislu
520591e6ea reset swap file chunk after uploading 2022-01-23 18:30:53 -08:00
chrislu
e185d90d24 2.87 2022-01-23 16:18:55 -08:00
chrislu
4a311c7f5e dedup local metadata subscribers 
fix https://github.com/chrislusf/seaweedfs/discussions/2542
 2022-01-23 16:14:22 -08:00
chrislu
bb6854b972 adjust log 2022-01-22 12:35:09 -08:00
chrislu
2bdd737971 rename file 2022-01-22 08:28:35 -08:00
chrislu
643bbbeb49 rename 2022-01-22 08:27:40 -08:00
chrislu
18ed06b420 use memory when under 16 chunks 2022-01-22 08:11:01 -08:00
chrislu
8e80f3cd65 move upload pipeline locking to a different file 2022-01-22 08:09:55 -08:00
chrislu
dde34fa99d rename 2022-01-22 08:08:01 -08:00
chrislu
a2aa542370 rename 2022-01-22 08:07:11 -08:00
chrislu
c376ccc5a5 swap file based random write large file upload 2022-01-22 08:06:53 -08:00
chrislu
b9ae16fbc5 fix memory allocation 2022-01-22 08:05:04 -08:00
chrislu
1ee828b768 refactor 
do not expose internal offset
 2022-01-22 06:34:29 -08:00
chrislu
02d0c12cdd rename 2022-01-22 06:00:10 -08:00
chrislu
5dea5c0449 refactor 2022-01-22 05:59:07 -08:00
chrislu
8aa6bf0bb9 refactoring 2022-01-22 05:40:10 -08:00
chrislu
d97bd54e63 just refactoring 2022-01-22 04:18:54 -08:00
chrislu
482014f9da rename file 2022-01-22 03:50:18 -08:00
chrislu
3b4a9addaf rename 2022-01-22 01:46:10 -08:00
chrislu
4acfc098e9 re-order 2022-01-22 01:43:14 -08:00
chrislu
e71dcfb3a6 add logging for memory allocation 2022-01-22 01:35:12 -08:00
chrislu
9d0f58c329 skip printing fs configuration 2022-01-21 13:29:47 -08:00
chrislu
ce2049cdb6 refactoring, move genFn before saveFn 2022-01-21 12:08:58 -08:00
chrislu
e47f63d159 enforce bucket quota 2022-01-21 02:34:42 -08:00
chrislu
6e57d8d0de s3: check bucket usage and adjust read only according to quota 2022-01-21 02:15:27 -08:00
chrislu
606667f205 able to configure the quota for a bucket 2022-01-21 01:42:20 -08:00
chrislu
b1063162b6 display bucket quota 2022-01-21 00:55:04 -08:00
chrislu
f103491912 s3: list bucket size from weed shell 2022-01-21 00:26:49 -08:00
chrislu
6c7135d77e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-21 00:17:06 -08:00
chrislu
ea57654e34 refactoring 2022-01-21 00:16:50 -08:00
Chris Lu
7c66f3b5fb
Merge pull request #2602 from kmlebedev/master_metrics 
master metricsHttpPort
 2022-01-20 09:26:25 -08:00
chrislu
b3e526ba95 url should be always using forward slash 2022-01-19 22:16:26 -08:00
Konstantin Lebedev
77c98b657e master metricsHttpPort 2022-01-19 21:43:22 +05:00
chrislu
77362700e1 S3: fail fast when "X-Amz-Copy-Source" is a folder 
fix #2593
 2022-01-18 12:04:40 -08:00
chrislu
05c3c3f56b Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-17 23:38:13 -08:00
chrislu
9b77f0054e 2.86 2022-01-17 23:38:03 -08:00
Chris Lu
ec254d8a89
Merge pull request #2597 from guol-fnst/gocql_to 
add gocql timeout setting
 2022-01-17 23:35:13 -08:00
chrislu
9274557552 keep dirty pages based on temp file 2022-01-17 23:23:49 -08:00
guol-fnst
da9540e666 add gocql timeout setting 2022-01-18 15:21:13 +08:00
chrislu
c87b8f4c30 S3: fail fast when "X-Amz-Copy-Source" is a folder 
fix https://github.com/chrislusf/seaweedfs/issues/2593
 2022-01-17 23:09:37 -08:00
chrislu
b2acfd75e9 ensure entry view cache is invalidated 2022-01-17 23:02:30 -08:00
chrislu
f4ad63528a wait for reading threads to complete before dropping sealed chunks 2022-01-17 22:24:44 -08:00
chrislu
0a3f95ca01 more logs 2022-01-17 20:41:00 -08:00
chrislu
b068bc291d testing with always resetting entry view cache 2022-01-17 20:07:01 -08:00
chrislu
047446d5ca remove extra async execution 2022-01-17 15:50:11 -08:00
chrislu
7bf7af971b more logs 2022-01-17 14:15:10 -08:00
chrislu
fc22071a2f more logs 2022-01-17 14:02:37 -08:00
chrislu
381f4e73a0 delete actual reference first 2022-01-17 13:56:47 -08:00
chrislu
0ba88596e8 invalidate filehandle entry view cache 2022-01-17 13:53:30 -08:00
chrislu
1734017ba1 add test 2022-01-17 13:40:41 -08:00
chrislu
da7f13e73e Revert "testing skip memory management" 
This reverts commit 6c908352cb.
 2022-01-17 03:21:31 -08:00
chrislu
6c908352cb testing skip memory management 2022-01-17 03:19:24 -08:00
chrislu
77d9993f38 remove unused variables 2022-01-17 03:19:11 -08:00
chrislu
f710d5ffca a little speed up 2022-01-17 03:19:00 -08:00
chrislu
fc0628c038 working 2022-01-17 01:53:56 -08:00
chrislu
1bd6d289d4 better locking on file handle 2022-01-15 05:45:29 -08:00
chrislu
2bfeb5d1c8 add filer to iam option 2022-01-15 03:37:52 -08:00
chrislu
b17c426e99 weed server: optionally start IAM service 
related to https://github.com/chrislusf/seaweedfs/issues/2560
 2022-01-13 22:49:49 -08:00
chrislu
3c8b74318e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-13 13:03:07 -08:00
chrislu
8907e6a40a add more help messages 2022-01-13 13:03:04 -08:00
banjiaojuhao
45e9c83421 padding zero for sparse file 2022-01-13 22:21:22 +08:00
chrislu
fe5b9e39cc POSIX: check permission when removing items 2022-01-13 02:07:39 -08:00
chrislu
1453263b63 remove dead code 2022-01-13 02:02:04 -08:00
chrislu
e69c374956 minor 2022-01-13 02:01:53 -08:00
chrislu
9b954dc0d4 adjust make file 2022-01-13 01:33:13 -08:00
chrislu
f2847f1266 POSIX: check deletion permission 2022-01-12 23:58:11 -08:00
chrislu
0c75f15062 POSIX: should not delete if a directory is not empty 2022-01-12 23:57:54 -08:00
chrislu
de27058d0b POSIX: differentiate device and char device 2022-01-12 21:45:38 -08:00
chrislu
d400a11832 POSIX: adjust source file ctime 
SeaweedFS uses mtime as ctime
 2022-01-12 21:45:18 -08:00
chrislu
b44f05a2d0 POSIX: change timestamp on each attribute change 2022-01-12 19:31:25 -08:00
chrislu
15c01d8b7f add some notes 2022-01-12 15:04:48 -08:00
chrislu
107a4884a8 shell: tighter memory allocation 2022-01-12 14:59:29 -08:00
chrislu
fec8428fd8 POSIX: different inode for same named different file types 2022-01-12 11:51:13 -08:00
chrislu
e82ad60122 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-12 11:07:49 -08:00
chrislu
caf0a3486b POSIX: adjust ctime for file truncate 2022-01-12 11:07:39 -08:00
Konstantin Lebedev
edb753ab4d https://github.com/chrislusf/seaweedfs/issues/2583 2022-01-12 16:04:59 +05:00
chrislu
adfd54e7c4 fix compilation 2022-01-12 01:24:24 -08:00
chrislu
6cc92817dc add logs for request mode 2022-01-12 01:13:19 -08:00
chrislu
826a7b307e master: remove hard coded filer settings in master.toml 
fix https://github.com/chrislusf/seaweedfs/issues/2529
 2022-01-12 01:11:25 -08:00
chrislu
cd1ad88f30 POSIX: check name is too long ENAMETOOLONG 2022-01-12 00:16:00 -08:00
chrislu
2dcb8cb93b POSIX: ensure file and directory inodes are different 
this is just an in memory representation.

POSIX wants different inode numbers for the same named file or directory.
 2022-01-11 23:44:48 -08:00
chrislu
5bb37d5905 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-11 23:29:12 -08:00
chrislu
10ecf80ca1 add a debug capability to list all metadata keys 2022-01-11 23:25:04 -08:00
Kyle Sanderson
9e012001be
filer.copy: don't crash when volume creation fails 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]

goroutine 7482 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x2)
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]

goroutine 7480 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x0)
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
 2022-01-11 22:22:39 -08:00
chrislu
1a7d5b5b5e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-11 12:24:56 -08:00
chrislu
41daecfdca Update mount_std.go 2022-01-11 12:23:12 -08:00
chrislu
2d0ccc4d34 add logs 2022-01-11 12:23:01 -08:00
Chris Lu
abe5da7d2c
Merge pull request #2575 from Radtoo/fix_paths2 
Fix paths2
 2022-01-11 12:04:30 -08:00
chrislu
b8fbf19e9a mount: rename follow POSIX 2022-01-11 03:23:03 -08:00
chrislu
6a12520a96 fix logging 2022-01-10 01:00:11 -08:00
chrislu
cbc055dc2b mount: file fsync 
fix https://github.com/chrislusf/seaweedfs/issues/2561
 2022-01-10 00:52:16 -08:00
chrislu
19555385f7 2.85 2022-01-09 19:30:23 -08:00
Radtoo
389002f195 Using positional arguments rather than option flag to enable better shell usage 2022-01-08 16:52:12 +01:00
Radtoo
fba1efb77a Now works with a single file too 
Parsing removed from doFixOneVolume

Needle init removed from runFix
 2022-01-08 16:31:53 +01:00
chrislu
110d5a5233 support fixing a collection of volumes, or volumes under one directory 2022-01-07 14:52:16 -08:00
chrislu
60dc450091 skip fixing read only volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2562
 2022-01-06 09:52:28 -08:00
chrislu
3df8f96117 avoid changing inode 2022-01-06 01:36:11 -08:00
chrislu
67b0645808 mount: need to change entry name after renaming 2022-01-05 21:27:41 -08:00
chrislu
4de060daa6 mount: skip special character in the filenames 
fix https://github.com/chrislusf/seaweedfs/issues/2559
 2022-01-05 03:57:24 -08:00
chrislu
e76105e2ab fix auth permission checking 2022-01-03 21:05:20 -08:00
chrislu
a7887166cf wildcard prefix to restrict access to directories in s3 bucket 
https://github.com/chrislusf/seaweedfs/discussions/2551
 2022-01-03 15:39:36 -08:00
chrislu
5799a20f71 2.84 2022-01-02 17:05:19 -08:00
Chris Lu
42c849e0df
Merge branch 'master' into metadata_follow_with_client_id 2022-01-02 01:07:30 -08:00
Chris Lu
9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158 
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
 2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst
c35660175d BUGFIX: ensure Authorization header is only added once 2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst
1cd3b6b4e1 BUGFIX: security.toml contained wrong keys 2021-12-31 22:05:41 +01:00
Sebastian Kurfuerst
10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client 
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
 2021-12-30 14:45:27 +01:00
chrislu
34742be029 remove duplicated metadata subscription in filer 
https://github.com/chrislusf/seaweedfs/issues/2545
 2021-12-30 01:51:52 -08:00
chrislu
5c87fcc6d2 add client id for all metadata listening clients 2021-12-30 00:23:57 -08:00
chrislu
fb434318e3 dynamically adjust connection timeout 
better fix for https://github.com/chrislusf/seaweedfs/issues/2541
 2021-12-29 22:44:39 -08:00
chrislu
5788bf2270 s3: increase timeout limit 
https://github.com/chrislusf/seaweedfs/issues/2541
 2021-12-29 22:21:02 -08:00
Sebastian Kurfuerst
fcc09cef6f Refactor: pass in claim type into security.DecodeJwt 2021-12-29 12:40:41 +01:00
Sebastian Kurfuerst
d156d410ef rename security.GenJwt to security.GenJwtForVolumeServer 2021-12-29 12:39:41 +01:00
Sebastian Kurfuerst
eda4c43a08 fix typo in error message 2021-12-29 12:38:14 +01:00
chrislu
498661e3bb mount: remove limits on number of parallel requests 2021-12-28 17:41:01 -08:00
chrislu
9a00c17555 reader: avoid wrong pattern detection due to lock waiting 2021-12-28 16:30:33 -08:00
chrislu
f7a6f6b4c0 if this is enabled, there are some "bus error" with git clone 2021-12-28 13:44:52 -08:00
chrislu
0da2dfd640 fuse: change to direct io mode 
before and after:

chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 4.534068 secs (236816430 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192  0.01s user 3.86s system 84% cpu 4.561 total
chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 3.824072 secs (280784948 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192  0.01s user 3.22s system 83% cpu 3.857 total
 2021-12-28 12:22:56 -08:00
chrislu
3fd4da34a4 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-28 01:10:38 -08:00
chrislu
2422556456 monitor write pattern: avoid timing due to locking 2021-12-28 01:10:35 -08:00
chenkai
47c30e3add filer list entries use context to break job 2021-12-28 15:03:41 +08:00
chrislu
80db8b13d8 bug: cleanup function was called twice 2021-12-27 20:53:02 -08:00
chrislu
67b723f74e Filer Server API support fsync 
fix https://github.com/chrislusf/seaweedfs/issues/2528
 2021-12-26 17:28:47 -08:00
chrislu
9f9ef1340c use streaming mode for long poll grpc calls 
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
 2021-12-26 00:15:03 -08:00
chrislu
c935b9669e 2.83 2021-12-25 01:01:34 -08:00
chrislu
eb4ad2546f use proper chunk size limit option 2021-12-24 22:52:18 -08:00
chrislu
41bbf320bb use 2MB chunk size. cache size is the wrong option 2021-12-24 22:50:19 -08:00
chrislu
982ea85d81 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-24 22:40:07 -08:00
chrislu
083d8e9ece add stream writer 
this should improve streaming write performance, which is common in many cases, e.g., copying large files.

This is additional to improved random read write operations: 3e69d19380...19084d8791
 2021-12-24 22:38:22 -08:00
Sebastian Kurfürst
6db49100d6 BUGFIX: add access.ui setting to scaffolded security.toml 
... The property is read here: b70cb3e0b2/weed/server/volume_server.go (L69)
 2021-12-24 13:59:04 +01:00
chrislu
255a1c7dcd refactor type names 2021-12-23 18:23:18 -08:00
chrislu
f77ca41769 refactor 2021-12-23 17:48:34 -08:00
chrislu
1d36884845 rename files 2021-12-23 17:47:58 -08:00
chrislu
2d1a1f5e03 rename variables and functions 2021-12-23 17:35:57 -08:00
chrislu
7bf48ee135 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-23 17:23:26 -08:00
chrislu
6de331b014 clean up 2021-12-23 17:23:21 -08:00
chrislu
032df784ed chunked file works now 2021-12-23 17:17:32 -08:00
banjiaojuhao
083bf3a137 filer server: add "datacenter, rack and datanode" for path specific configuration 2021-12-23 23:25:05 +08:00
chrislu
c2aad1c7ff detect non streaming mode on first write request 2021-12-22 17:20:44 -08:00
chrislu
b541e39a2c fix tests 2021-12-22 16:17:30 -08:00
chrislu
2bc6fa90ff Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-22 16:05:43 -08:00
chrislu
0ec7bc6710 detect non streaming mode on the first read 2021-12-22 16:05:38 -08:00
chrislu
4c1368d621 fix test 2021-12-22 16:05:08 -08:00
Chris Lu
4e73705533
Merge pull request #2530 from banjiaojuhao/filer-upload-file-to-node 
filer server: allow upload file to specific dataNode
 2021-12-22 12:49:15 -08:00
banjiaojuhao
08336be92e filer server: allow upload file to specific dataNode 2021-12-22 21:57:26 +08:00
chrislu
7b78fc72b0 add page chunk interval list 2021-12-22 02:53:33 -08:00
chrislu
b7cd52636b Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-21 22:24:43 -08:00
chrislu
3981d65b68 remove println 2021-12-21 22:24:38 -08:00
chrislu
9a73319b45 mount: different write strategy for streaming write and random write 2021-12-21 17:28:55 -08:00
Chris Lu
b0665a15f4
Merge pull request #2527 from banjiaojuhao/master-assign-by-datanode 2021-12-21 08:56:51 -08:00
banjiaojuhao
dda6b90d25 assign fileId according to DataNode with empty DataCenter and Rack 2021-12-21 17:28:33 +08:00
chrislu
4b8dcff448 reverting default admin scripts 
fix https://github.com/chrislusf/seaweedfs/issues/2525

this new default value was introduced in 2.80
this affects production environments, e.g., EC is not desired, volume balancing is not preferred, etc.
 2021-12-20 13:34:57 -08:00
chrislu
b21a67bbe6 add writer pattern object for later use 2021-12-20 11:53:48 -08:00
chrislu
4fd29dad86 remove writeOnly flag 2021-12-20 01:11:43 -08:00
chrislu
bc96682760 refactor, change file locations 2021-12-20 01:02:23 -08:00
chrislu
866c2657f0 avoid FUSE cache only for the first 512 bytes 2021-12-19 23:13:36 -08:00
chrislu
0cb9036f66 mount: only cache the first chunk on stream read 2021-12-19 23:06:03 -08:00
chrislu
a152f17937 mount: improve read performance on random reads 2021-12-19 22:43:14 -08:00
chrislu
85c526c583 s3: bind to a specific IP 
fix https://github.com/chrislusf/seaweedfs/issues/2516
 2021-12-17 11:34:37 -08:00
chrislu
5eacff9d4f log message adds server name 
address https://github.com/chrislusf/seaweedfs/issues/2514#issuecomment-995925733
 2021-12-16 10:46:26 -08:00
chrislu
50ddd8c8e2 remove debug messages 
fix https://github.com/chrislusf/seaweedfs/issues/2514
 2021-12-16 00:58:15 -08:00
chrislu
7210558c7b s3: pass through s3 presigned headers 
fix https://github.com/chrislusf/seaweedfs/discussions/2502
 2021-12-15 13:18:53 -08:00
chrislu
bf4d7affc0 gateway to remote object store: skip replicating multipart upload part files 
fix https://github.com/chrislusf/seaweedfs/issues/2509
 2021-12-14 19:48:31 -08:00
chrislu
316f326464 add more help message 2021-12-13 13:14:36 -08:00
Konstantin Lebedev
969f513265 disable audit log for s3 statusHandler 2021-12-13 16:08:19 +05:00
Chris Lu
ab0c6a84bf
Merge pull request #2503 from kmlebedev/audit_log_nonblocking 
Audit log force async
 2021-12-13 00:47:16 -08:00
Konstantin Lebedev
34779e8f38 force enable asynchronous I/O sending events to Fluentd 2021-12-13 13:39:39 +05:00
chrislu
c3b73ec23b 2.82 2021-12-12 23:25:24 -08:00
chrislu
7bf14452f7 s3: handle Response-Content-Disposition 2021-12-12 23:05:40 -08:00
chrislu
94e5c0a454 skip purging from ec volumes 2021-12-12 22:55:27 -08:00
chrislu
a2d3f89c7b add lock messages 2021-12-10 13:24:38 -08:00
chrislu
991a3dca0d rename file 2021-12-10 13:11:00 -08:00
kmlebedev
4f98553ba9 audit log SignatureVersion 2021-12-10 19:40:32 +05:00
chrislu
2d6fcdf83a add delay if need to resume from disk 2021-12-09 13:13:05 -08:00
liubaojiang
c7abc34dbb rename operation returns the correct old parent path 2021-12-10 00:02:57 +08:00
Konstantin Lebedev
98251fe16a non blocking audit log 2021-12-09 19:47:16 +05:00
chrislu
40ee9b4d7b sleep between the gap of each execution 
related to https://github.com/chrislusf/seaweedfs/issues/2476
 2021-12-07 11:44:08 -08:00
Chris Lu
2ba08afed1
Merge pull request #2498 from kmlebedev/s3_audit_log 2021-12-07 09:35:48 -08:00
Konstantin Lebedev
10678cde81 audit log config 2021-12-07 18:20:52 +05:00
Konstantin Lebedev
4ec8715f20 audit log 2021-12-07 12:15:48 +05:00
Konstantin Lebedev
3ac48cd540 audit log 2021-12-07 12:13:21 +05:00
chrislu
7f0a97c7b6 shell: volume.list supports different verbosity level 2021-12-05 21:54:40 -08:00
chrislu
5ea9715721 2.81 
also sync java client version to SeaweedFS version
 2021-12-05 18:05:24 -08:00
chrislu
e6c026db65 volume.fix.replication: fix misplaced volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2416
 2021-12-05 16:56:25 -08:00
chrislu
53e2dee177 [volume.check.disk] was using the wrong source and target locations 
fix https://github.com/chrislusf/seaweedfs/issues/2268
 2021-12-05 14:32:04 -08:00
chrislu
c146c76d10 avoid creating the same bucket with a different randomized name 
related to https://github.com/chrislusf/seaweedfs/issues/2492
 2021-12-05 13:06:41 -08:00
chrislu
42d97a3442 adjust randomized bucket name 2021-12-05 12:36:58 -08:00
chrislu
488afa5002 volume: load volume can optionally be skipped, if ec volume exists 
fix https://github.com/chrislusf/seaweedfs/issues/2489
 2021-12-05 02:28:52 -08:00
chrislu
59e58c4b23 volume: fix loading old volume format 
fix https://github.com/chrislusf/seaweedfs/issues/2487
 2021-12-05 01:06:01 -08:00
chrislu
b70cb3e0b2 upgrade protoc to 3.17.3 
$brew install protobuf
$ protoc --version
libprotoc 3.17.3
$ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26
$ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1
 2021-12-05 00:42:25 -08:00
chrislu
5c3b783310 s3: copy object API needs to escape special characters 
fix https://github.com/chrislusf/seaweedfs/issues/2482
 2021-12-04 23:24:53 -08:00
Bl1tz23
83bffca7e6 fix MongodbStore.ListDirectoryEntries panic on find failure 2021-12-03 13:58:37 +03:00
Tanmoy Majumdar
ea09fb477a return ' shouldRetry=true' so that filer can retry the failed chunk 2021-12-03 11:54:20 +06:00
Chris Lu
689f5513a9 redis3 supports sentinel 2021-11-29 01:09:51 -08:00
Chris Lu
7227cfddf5 2.80 2021-11-29 00:57:08 -08:00
Chris Lu
3d7390302d add s3.clean.uploads -timeAgo=24h 2021-11-29 00:49:49 -08:00
Chris Lu
2f72c24498 skip the rest logic 2021-11-29 00:32:21 -08:00
Chris Lu
88ff8fc27b ensure uploaded chunks are deleted on error 2021-11-29 00:28:26 -08:00
Chris Lu
bea26549f2 avoid empty extended attribute value 2021-11-28 23:50:52 -08:00
Chris Lu
ce2af0082e revert 2021-11-28 23:35:22 -08:00
Chris Lu
1c9f3c7ac0 read deleted chunks when replcating data 2021-11-28 23:34:34 -08:00
Chris Lu
96c66ca2aa read deleted chunks when replicating data 2021-11-28 23:33:03 -08:00
Chris Lu
f3a334965d Update command_remote_uncache.go 
minor
 2021-11-28 23:10:32 -08:00
Chris Lu
9ccfc1cfee Update command_volume_fsck.go 
minor
 2021-11-28 22:39:24 -08:00
Chris Lu
71a94267c9 Update command_volume_fsck.go 
remove unused code
 2021-11-28 22:37:11 -08:00
Chris Lu
ad16221a35 adjust error log 2021-11-28 22:06:17 -08:00
Chris Lu
cf1586a34d add logs for writing to remote file 2021-11-27 22:09:23 -08:00
Chris Lu
3a19eea97c allocate memory by slabs 2021-11-27 12:13:00 -08:00
limd
8805c04128 fix redis2 sentinel config example 2021-11-25 19:20:02 +08:00
limd
ec03f22cc3 Merge remote-tracking branch 'origin/master' 2021-11-25 16:07:14 +08:00
limd
220797bd71 support redis sentinel 2021-11-25 15:57:03 +08:00
Chris Lu
6c27845be0 add retries when writing to remote s3 
fix https://github.com/chrislusf/seaweedfs/issues/2465
 2021-11-22 21:48:04 -08:00
Chris Lu
f3c789d662 2.79 2021-11-21 18:40:24 -08:00
Konstantin Lebedev
40abae7caa Avoid forbiddening for HeadBucketHandler if owners are specified 
https://github.com/chrislusf/seaweedfs/issues/2434
 2021-11-17 14:42:24 +05:00
Chris Lu
3d87aa767d fix same dc and other dc 2021-11-16 09:14:01 -08:00
Chris Lu
a0ef6e3611 prevent nil response 
fix https://github.com/chrislusf/seaweedfs/issues/2452
 2021-11-15 08:46:01 -08:00
Chris Lu
100c654ec3 2.78 2021-11-14 23:29:59 -08:00
Chris Lu
7bf891c00a randomize same-dc servers and other-dc servers 2021-11-12 11:30:11 -08:00
Chris Lu
1f75f1f9dc filer: fix mysql2 SQL template 2021-11-11 22:28:28 -08:00
Chris Lu
3abbaccb70 filer: fix mysql command to upsert 2021-11-11 22:27:13 -08:00
user
9668b15f38 Filtering by volume id is prioritized. 2021-11-11 18:18:56 +09:00
user
563a74a9eb Volume filter by collection pattern added. 2021-11-11 18:01:47 +09:00
user
dbb8003ce3 Volume filter function added. 2021-11-11 17:45:17 +09:00
user
c387fe957b Unused parameter removed. 2021-11-11 16:08:41 +09:00
Konstantin Lebedev
42f4aa824b force overwrite s3-is-admin 
https://github.com/chrislusf/seaweedfs/issues/2433
 2021-11-11 11:34:01 +05:00
Chris Lu
c77c0de418 make install 2021-11-09 12:34:57 -08:00
Chris Lu
a7eee8b14b
Merge pull request #2427 from Bl1tz23/master 
Fix: potencial iam identities data race
 2021-11-09 08:53:48 -08:00
Chris Lu
1d4a61af5e add brokers 2021-11-09 08:50:55 -08:00
Bl1tz23
c683409e92 s3: add RWMutex to iam, use RLock for concurrent reading 2021-11-09 18:11:06 +03:00
Bl1tz23
5018b22f36 s3: fix potencial iam identities data race 2021-11-09 12:19:50 +03:00
Chris Lu
59d1435d65 s3: avoid possible attacks by version 2021-11-08 17:47:56 -08:00
Chris Lu
5c99b8acc3 rename variables 2021-11-08 17:47:56 -08:00
Chris Lu
00fdab1284 adjust client name 2021-11-08 17:47:56 -08:00
Chris Lu
4729a57cc0 use constants 2021-11-08 17:47:56 -08:00
Chris Lu
d9dd72ea56 rename pacakge 2021-11-08 17:47:56 -08:00
Chris Lu
5cf332357b 2.77 2021-11-07 13:52:45 -08:00
Chris Lu
244841f080 adjust logs 2021-11-07 13:27:57 -08:00
Chris Lu
6e999f1176 s3: fix ListObject if more than 10000 objects 2021-11-07 12:39:36 -08:00
Chris Lu
a8b0f8864d fix help message 2021-11-07 12:38:35 -08:00
Chris Lu
55973e8572 log unknown access key 2021-11-07 12:37:46 -08:00
Chris Lu
c4e22b5a9a filer: deprecate "-peers" option 2021-11-06 14:36:45 -07:00
Chris Lu
751a7073e3 Merge branch 'flexible_filer_addition' 2021-11-06 14:26:26 -07:00
Chris Lu
e0fc2898e9 auto updated filer peer list 2021-11-06 14:23:35 -07:00
Chris Lu
04663c3611 remote.mount: print out metadata sync errors 2021-11-06 11:29:50 -07:00
Chris Lu
330d1fde7f send peers info to filers 2021-11-06 11:29:50 -07:00
Chris Lu
84bb8e7365 send peers info to filers 2021-11-06 04:07:38 -07:00
Chris Lu
4b9c42996a refactor grpc API 2021-11-05 18:11:40 -07:00
Chris Lu
5ea86ef1da Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()" 
This reverts commit af71ae11aa.
 2021-11-05 17:52:15 -07:00
Chris Lu
314c32514b add command to list filers 2021-11-05 17:39:41 -07:00
Chris Lu
7348d2ae63 randomize traffic if replicated 2021-11-05 00:03:13 -07:00
Chris Lu
00ae965d8d randomize a bit for ec shards distribution 2021-11-04 09:23:40 -07:00
Chris Lu
77f90ae288 add leader election in master 2021-11-04 00:54:38 -07:00
Chris Lu
35c37562bc check auth only when enabled 2021-11-03 12:11:36 -07:00
Chris Lu
3c245c69d3 handle possible race condition 2021-11-03 01:36:39 -07:00
Chris Lu
af71ae11aa master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates() 2021-11-03 01:09:48 -07:00
Chris Lu
ab97b17e62 better printout 2021-11-02 23:45:47 -07:00
Chris Lu
e66865a8c5 adjust master client log level 2021-11-02 23:45:28 -07:00
Chris Lu
0c8dea9de8 go fmt 2021-11-02 23:39:16 -07:00
Chris Lu
5160eb08f7 shell: optionally read filer address from master 2021-11-02 23:38:45 -07:00
Chris Lu
18bfbf62fc add one unit test 2021-11-02 23:36:43 -07:00
Chris Lu
e1ab8b01d0 s3: adjust permission for HEAD bucket operation 
fix https://github.com/chrislusf/seaweedfs/issues/2417#issuecomment-958391856
 2021-11-02 16:28:55 -07:00
Chris Lu
b25661c6df s3: skip permission checking for creating bucket if the bucket already exists 
fix https://github.com/chrislusf/seaweedfs/issues/2417

Rclone was trying to create the bucket even though the bucket already exists.
 2021-11-02 13:13:36 -07:00
Chris Lu
1b90d60765 filer: paginate large directories to delete file chunks 2021-11-02 01:04:50 -07:00
Chris Lu
794375ca0a adjust help message since both fullPercent and quietFor are needed. 2021-11-01 17:22:47 -07:00
Chris Lu
fc9e246592 2.76 2021-10-31 18:08:28 -07:00
Chris Lu
9cf756b188 S3: support CORS 
related to https://github.com/chrislusf/seaweedfs/issues/1271
 2021-10-31 18:06:43 -07:00
Chris Lu
f17fa400d5 refactoring 2021-10-31 18:05:34 -07:00
Chris Lu
006b4ec0f8 refactoring 2021-10-31 18:02:08 -07:00
Chris Lu
edbf6d297b filer.meta.tail: add example to send metadata to elastic search 2021-10-31 18:01:33 -07:00
Chris Lu
24858507cc rename API to avoid confusion 2021-10-30 19:27:25 -07:00
Chris Lu
f5824f369c avoid reporting error when it is a user request error 2021-10-29 14:14:29 -07:00
Chris Lu
c857cc7286 cloud tier: remove tagging since not all s3 vendors support this 2021-10-29 12:39:19 -07:00
Chris Lu
d04cdcf40d s3 header add user agent 2021-10-29 12:28:24 -07:00
Konstantin Lebedev
dc3ae5e8a6 DeleteBucketLifecycleConfiguration return 204 2021-10-28 19:34:37 +05:00
Konstantin Lebedev
2afb5a13af fix DeleteBucketLifecycleConfiguration 2021-10-28 18:30:33 +05:00
Chris Lu
2e76834e4d filer store: redis2 fix wrong pagination 2021-10-27 23:46:25 -07:00
Chris Lu
54b6e0f3fd adjust logs 2021-10-27 23:46:07 -07:00
Chris Lu
900e888695 filer: avoid possible prefixed pagination loop on unsupported filer stores 2021-10-27 23:45:48 -07:00
Chris Lu
d774fa6c9a rename variable 2021-10-25 14:39:20 -07:00
Chris Lu
2539ba0b62 fix compilation 2021-10-25 14:38:48 -07:00
Chris Lu
5f2d7c1589 erasure coding: skip erasure coding if less than recommended 4 nodes 2021-10-25 14:38:11 -07:00
Chris Lu
c9d3fb4a30 2.75 2021-10-24 18:15:59 -07:00
Chris Lu
5435027ff0 volume copy: stream out copying progress and avoid grpc request timeout 
fix https://github.com/chrislusf/seaweedfs/issues/2386
 2021-10-24 02:52:56 -07:00
Chris Lu
3be3c17f59 volume vacuum: avoid timeout with streaming progress report 
fix https://github.com/chrislusf/seaweedfs/issues/2396
 2021-10-24 01:55:34 -07:00
Konstantin Lebedev
235329a92a fix Tagging add xmlns 2021-10-20 19:40:22 +05:00
Konstantin Lebedev
dc7e525cb9 fix Tagging test 2021-10-20 19:12:00 +05:00
Konstantin Lebedev
0b834600d5 fix PutObjectTaggingHandler Unmarshal 2021-10-20 17:58:06 +05:00
Konstantin Lebedev
c78220a7f2 fix object tagging https://github.com/chrislusf/seaweedfs/issues/2389 2021-10-20 16:01:06 +05:00
Chris Lu
182f43ae5f 2.74 2021-10-18 14:23:54 -07:00
Chris Lu
309f46e3af S3: fix upload limit if the size is more than 8GB 2021-10-18 14:14:18 -07:00
Chris Lu
cd4fa7561b 2.73 2021-10-18 10:47:48 -07:00
Chris Lu
a6bb509cb7 adjust mime detection 2021-10-18 10:47:39 -07:00
Konstantin Lebedev
dd2cc1acaf fix https://github.com/chrislusf/seaweedfs/issues/2387 2021-10-18 16:27:57 +05:00
Chris Lu
97c963bac9 2.72 2021-10-17 17:40:27 -07:00
Chris Lu
3833dac3f7 continue to read from memory if there is no flush 2021-10-17 13:53:04 -07:00
Chris Lu
29fa1b9fdc add logs 2021-10-17 13:50:34 -07:00
Chris Lu
cca62fdb30 mount: streaming renaming folders 2021-10-17 04:22:42 -07:00
Chris Lu
004e56c1a6 save updated lastTsNs 2021-10-17 02:01:20 -07:00
Chris Lu
71a8c80f6f s3: continuously listen to file updates 
fix https://github.com/chrislusf/seaweedfs/issues/2342
 2021-10-17 01:04:19 -07:00
Chris Lu
93bb7869b8 Revert "mount: fix renaming a deep directory with unvisited directories" 
This reverts commit 0ccdb937bb.
 2021-10-16 23:52:30 -07:00
Chris Lu
8e2c9713a3 turn on new faster algorithm to translate into visible chunks 2021-10-16 23:35:20 -07:00
Chris Lu
e7d1f183cc ensure file id is populated 2021-10-16 23:34:27 -07:00
Chris Lu
fc1694b321 add tests 2021-10-16 23:34:12 -07:00
Chris Lu
0ccdb937bb mount: fix renaming a deep directory with unvisited directories 2021-10-16 23:33:45 -07:00
Chris Lu
b9a2efd69b temporarily reverting 2021-10-16 16:29:10 -07:00
Chris Lu
8a124ef9fc Revert "remove deprecated code" 
This reverts commit de7688c539.
 2021-10-16 16:09:33 -07:00
Chris Lu
93624635b4 Revert "go fmt" 
This reverts commit 6c704eb2ba.
 2021-10-16 16:09:30 -07:00
Chris Lu
6c704eb2ba go fmt 2021-10-16 16:06:18 -07:00
Chris Lu
de7688c539 remove deprecated code 2021-10-16 16:06:12 -07:00
Chris Lu
7336990639 faster file read for large files 2021-10-16 16:03:16 -07:00
Chris Lu
8965a53c4d add warning error 2021-10-16 15:57:30 -07:00
Chris Lu
8e3075d051 ensure file entry chunks have populated the file id 2021-10-16 01:05:48 -07:00
Chris Lu
822ee2475b remove unnecessary file 2021-10-16 01:05:06 -07:00
Konstantin Lebedev
8de7915950 s3 mime detect 2021-10-14 15:03:11 +05:00
Chris Lu
5fd4b05c5e
Merge pull request #2381 from Juneezee/deprecate-ioutil 
refactor: move from io/ioutil to io and os package
 2021-10-13 22:38:58 -07:00
Chris Lu
46a09c6074 adjust test 2021-10-13 22:38:47 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-14 12:27:58 +08:00
Chris Lu
4cbd390fbe test: add fail message 2021-10-13 20:42:20 -07:00
Chris Lu
b693a8d6b9
Merge pull request #2380 from kmlebedev/bucket_policy 
Bucket policy
 2021-10-13 15:29:53 -07:00
Konstantin Lebedev
9d6ffa0ea1 GetBucketLifecycleConfigurationHandler 2021-10-14 01:35:33 +05:00
Konstantin Lebedev
458145425e S3 MultipartUpload pass contentType to meta 2021-10-12 17:14:54 +05:00
Chris Lu
de1ec050e8 fier/s3: save "Content-Disposition" to extended properties 
related to https://github.com/chrislusf/seaweedfs/issues/2371
 2021-10-12 00:04:28 -07:00
Chris Lu
34f764007f escape file name only when necessary 2021-10-11 23:33:45 -07:00
Chris Lu
8e9273db99 s3: use "response-content-disposition" to overwrite default content-disposition 
fix one part of https://github.com/chrislusf/seaweedfs/issues/2371

see https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
 2021-10-11 23:29:52 -07:00
Chris Lu
f0b928ff5e go fmt 2021-10-11 23:23:46 -07:00
Chris Lu
3afa451cdc volume: find a non-empty offset when binary searching by timestamp 2021-10-11 22:00:41 -07:00
Chris Lu
b530f12327 volume: find a non-empty offset when binary searching by timestamp 
fix https://github.com/chrislusf/seaweedfs/issues/2364
 2021-10-11 16:11:50 -07:00
Konstantin Lebedev
be4b3ed509 AclHandlers 2021-10-11 15:03:56 +05:00
Chris Lu
e6ef7b2387 return error early 
fix https://github.com/chrislusf/seaweedfs/issues/2370
 2021-10-11 01:24:30 -07:00
Chris Lu
3d586be552 2.71 2021-10-10 22:40:44 -07:00
Chris Lu
f4676824a7 fix test code compilation 2021-10-10 22:14:13 -07:00
Chris Lu
8205166009 disable testing with redis server 
--- FAIL: TestNameList (0.00s)
panic: exec: "redis-server": executable file not found in $PATH [recovered]
	panic: exec: "redis-server": executable file not found in $PATH

goroutine 37 [running]:
testing.tRunner.func1.2({0xde2f80, 0xc0003da160})
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1209 +0x24e
testing.tRunner.func1()
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1212 +0x218
panic({0xde2f80, 0xc0003da160})
	/opt/hostedtoolcache/go/1.17.1/x64/src/runtime/panic.go:1038 +0x215
github.com/chrislusf/seaweedfs/weed/filer/redis3.TestNameList(0x407c59)
	/home/runner/work/seaweedfs/seaweedfs/weed/filer/redis3/kv_directory_children_test.go:45 +0x376
testing.tRunner(0xc0003e2680, 0xf57f48)
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1259 +0x102
created by testing.(*T).Run
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1306 +0x35a
 2021-10-10 21:57:55 -07:00
Chris Lu
3e2acf677c removing tikv to resolve "go mod tidy" problem 
tikv is causing "go mod tidy" problem. Need to resolve this before adding tikv back.

go mod tidy
go: finding module for package github.com/coreos/etcd/clientv3/balancer/picker
go: finding module for package cloud.google.com/go/kms/apiv1
go: finding module for package github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
go: finding module for package google.golang.org/grpc/naming
go: finding module for package github.com/coreos/etcd/clientv3/credentials
go: finding module for package github.com/coreos/etcd/clientv3/balancer
go: finding module for package github.com/d4l3k/messagediff
go: found github.com/coreos/etcd/clientv3/balancer in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/picker in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/resolver/endpoint in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/credentials in github.com/coreos/etcd v3.3.26+incompatible
go: found cloud.google.com/go/kms/apiv1 in cloud.google.com/go/kms v1.0.0
go: found github.com/d4l3k/messagediff in github.com/d4l3k/messagediff v1.2.1
go: finding module for package google.golang.org/grpc/naming
github.com/chrislusf/seaweedfs/weed/filer/tikv imports
	github.com/tikv/client-go/v2/tikv imports
	go.etcd.io/etcd/clientv3 tested by
	go.etcd.io/etcd/clientv3.test imports
	github.com/coreos/etcd/integration imports
	github.com/coreos/etcd/proxy/grpcproxy imports
	google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
 2021-10-10 19:27:02 -07:00
Chris Lu
bf218cd59d removing etcd sequencer 
causing go mod tidy problem. If anyone wants this, please help to resolve this first.

github.com/chrislusf/seaweedfs/weed/sequence imports
	go.etcd.io/etcd/client tested by
	go.etcd.io/etcd/client.test imports
	github.com/coreos/etcd/integration imports
	github.com/coreos/etcd/proxy/grpcproxy imports
	google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
 2021-10-10 19:10:46 -07:00
Chris Lu
17e08a16f4 fix redis3 deletion 2021-10-09 04:54:48 -07:00
Chris Lu
ce1efeb0eb adds more error message 2021-10-09 04:54:14 -07:00
Chris Lu
e4830bd93d go fmt 2021-10-07 21:13:31 -07:00
Chris Lu
0a856241fe avoid int bigger than math.MaxInt32 
fix https://github.com/chrislusf/seaweedfs/issues/2363
 2021-10-07 21:12:57 -07:00
Chris Lu
d688e10ed1 do not try to compress if not sure about the file content 2021-10-07 13:29:00 -07:00
Chris Lu
332f5ad3a8 revert temporary changes that disabled compression 
fix https://github.com/chrislusf/seaweedfs/issues/2362
 2021-10-07 13:24:16 -07:00
Chris Lu
f3d8232e14 reduce one redis lookup on hot path 2021-10-06 22:01:19 -07:00
Chris Lu
2336a397dc use pipeline to save some time 2021-10-06 20:40:18 -07:00
Chris Lu
371fead8a5 redis3 using redis native sorted set 2021-10-06 18:18:24 -07:00
Chris Lu
8668d49c9d test with real redis 
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkNameList-12    	    1789	    760599 ns/op
BenchmarkRedis-12       	   17539	     64122 ns/op
PASS
 2021-10-06 01:25:37 -07:00
Chris Lu
6b31f3c97a add benchmark test 2021-10-06 00:37:57 -07:00
Chris Lu
893f0587b1 redis3 adds distributed locking 2021-10-06 00:03:54 -07:00
Chris Lu
f0d1e7bd05 skip ec volumes when loading normal volumes 2021-10-05 02:31:44 -07:00
Chris Lu
332d49432d reduce concurrent volume grow requests 2021-10-05 01:58:30 -07:00
Chris Lu
96119eab00 refactor 2021-10-05 00:40:04 -07:00
Chris Lu
8a66306064 calculate disk usage in case of race condition 
related to https://github.com/chrislusf/seaweedfs/issues/2357
 2021-10-04 23:32:07 -07:00
Chris Lu
4ed2994555 use tsMemory to determine whether read from disk or memory 
remove lastFlushTime
 2021-10-04 16:02:56 -07:00
Chris Lu
6a030547a2 server: remove peer check if not starting master 
more fix https://github.com/chrislusf/seaweedfs/issues/2352
 2021-10-04 03:27:10 -07:00
Chris Lu
62c2732fd1 add back logic to check master peers 
fix https://github.com/chrislusf/seaweedfs/issues/2352
 2021-10-04 02:51:26 -07:00
Chris Lu
d4bb16e20e
Merge pull request #2354 from chrislusf/bptree 
Add Redis3
 2021-10-04 02:43:15 -07:00
Chris Lu
947add39e6 clean up *SkipListElementReference loaded from Redis 2021-10-04 02:31:38 -07:00
Chris Lu
513fed323a SkipListElementReference can be an empty object 2021-10-04 02:30:44 -07:00
Chris Lu
280ab7f95c add test 2021-10-04 02:30:24 -07:00
Chris Lu
04662126bb add redis3 2021-10-04 01:04:27 -07:00
Chris Lu
2b9aab3442 use 1000 per batch 2021-10-04 01:03:40 -07:00
Chris Lu
366f522a2d add redis3 2021-10-04 01:01:31 -07:00
Chris Lu
ba7fbac07f rename 2021-10-03 19:23:34 -07:00
Chris Lu
e6196cdc50 add name list 2021-10-03 17:54:25 -07:00
Chris Lu
a481c4a45e return previous element if visited 2021-10-03 13:50:52 -07:00
Chris Lu
22d8684e88 refactor out listStore 2021-10-03 02:19:21 -07:00
Chris Lu
d343b0db57 update value 2021-10-03 01:15:14 -07:00
Chris Lu
4f50f8c2ca insert key and value 2021-10-03 01:07:35 -07:00