mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-04 02:39:01 +08:00
Code Issues Packages Projects Releases Wiki Activity
6,158 Commits 32 Branches 296 Tags 149 MiB
2d805beed4
Commit Graph

4017 Commits

Author SHA1 Message Date
Chris Lu
07f20155fd small optimization 2021-07-01 01:19:31 -07:00
danielflira
a66112c9d2 configurable fusermount path 2021-07-01 00:07:54 -03:00
Kevin Liu
253b4ed898
remote -> proxy in server to match volume flags 
noticed this was missed in https://github.com/chrislusf/seaweedfs/pull/2168
 2021-06-30 13:05:45 -07:00
zhangsong
20d33ae025 add proxy mode to read non-local volumes 2021-06-30 18:33:18 +08:00
zhangsong
7566782c2e add proxy mode to read non-local volumes 2021-06-30 17:28:37 +08:00
陈杨文
6f683e6572
also failed on mips64 
alpine support mips64, will try to build on next version
 2021-06-30 04:36:00 +08:00
Chris Lu
41db292332 skip s390 ppc64le 
due to https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/22703
 2021-06-29 10:06:38 -07:00
Chris Lu
3668d10664 range query for mp4 video play 
fix https://github.com/chrislusf/seaweedfs/issues/2156
 2021-06-29 02:13:29 -07:00
Chris Lu
24e11d1e90 look back when adding to sorted values 
look back when adding to sorted values, before adding it to overflow
 2021-06-28 22:46:49 -07:00
Chris Lu
fc8dd58aea volume: large_volume version has bug when using in memory index 
fix https://github.com/chrislusf/seaweedfs/issues/2162
 2021-06-28 15:48:07 -07:00
Chris Lu
a2979aa051 2.56 2021-06-27 23:33:45 -07:00
Chris Lu
08377fecb8 fixes 2021-06-27 23:32:57 -07:00
Chris Lu
17477b37d5 sleep before re-reading the messages 
If there are no more metadata changes and the client disconnects, it would go into a busy loop without this fix.
 2021-06-27 06:31:04 -07:00
Chris Lu
c764596f96 filer: slow metadata topic read may lose some change events 
fix https://github.com/chrislusf/seaweedfs/issues/2117
 2021-06-27 05:54:16 -07:00
Chris Lu
cc7714fdbe logging changes to debug 2021-06-27 05:54:16 -07:00
thephoenixofthevoid
8d70ba2eaa Fix: Loosing environment variables at StartProcess 2021-06-26 15:20:27 +03:00
Chris Lu
1cac2f2278 shell: volume.fsck checks missing chunks in filer 
fix https://github.com/chrislusf/seaweedfs/issues/2154
 2021-06-24 23:56:24 -07:00
Chris Lu
9dd09bbb33 refactor 2021-06-24 17:22:53 -07:00
Chris Lu
c2e0a75c1f adjust logs 2021-06-24 12:46:00 -07:00
Chris Lu
78b1fb921c adjust log level 2021-06-23 20:59:54 -07:00
Chris Lu
05af54ad10 2.55 
trigger migrated travis build
 2021-06-23 00:41:04 -07:00
Chris Lu
d4b0dcf442
Merge pull request #2147 from jonaz/process_metrics 
Add process metrics of weed itself
 2021-06-22 04:13:17 -07:00
Jonas Falck
829b195084 Add process metrics of weed itself 2021-06-22 13:09:42 +02:00
Chris Lu
d474ce6fe3 master: avoid repeated leader redirection 
fix https://github.com/chrislusf/seaweedfs/issues/2146
 2021-06-21 22:56:07 -07:00
danielflira
f1d207a0fe start weed with mount in background 2021-06-20 02:48:46 -03:00
Chris Lu
b3eb4fecc7 2.54 2021-06-19 03:48:15 -07:00
Chris Lu
f24bb9e688 mount: fix for deletion stopped working since 2.53 
fix https://github.com/chrislusf/seaweedfs/issues/2138

due to 4d0cbd2700
 2021-06-19 03:46:39 -07:00
Chris Lu
88d52adfdd remove unused fields 2021-06-18 15:35:22 -07:00
Chris Lu
56eb522b13 fix stats when a collection is deleted 2021-06-15 21:11:31 -07:00
Chris Lu
055374a50b FUSE: skip flushing if file is deleted 
related to https://github.com/chrislusf/seaweedfs/issues/2110
 2021-06-15 12:45:23 -07:00
Konstantin Lebedev
6aa1a56ec8 avoid crashes Galera Cluster 
https://github.com/chrislusf/seaweedfs/issues/2125
 2021-06-15 18:12:39 +05:00
Chris Lu
ebe971da2e
Merge pull request #2127 from nagy/check-history-error 
Check for history file errors as well
 2021-06-15 03:29:21 -07:00
Chris Lu
7a81caa31e 2.53 2021-06-13 17:12:34 -07:00
Chris Lu
4d0cbd2700 skip cookie checking if from grpc api 2021-06-13 16:16:11 -07:00
Chris Lu
7225cb4ac5 add block and mutex profiling 2021-06-13 16:15:54 -07:00
Chris Lu
28a4a1f8d6 fix for mysql2 postgres2 on fast dropping buckets 2021-06-13 07:31:56 -07:00
Chris Lu
1e76fc994a filer.copy: zero fileSize for directories 2021-06-13 07:09:06 -07:00
Chris Lu
ed6aa13520 minor 2021-06-12 18:36:25 -07:00
Chris Lu
ee6c67682c minor 2021-06-12 02:52:41 -07:00
Chris Lu
9357911a95 remove all bucket metadata 
fix https://github.com/chrislusf/seaweedfs/issues/2118
 2021-06-10 23:37:54 -07:00
Chris Lu
b71c3cfba4 avoid possible empty object 2021-06-10 22:17:53 -07:00
Chris Lu
310e31424e adjust the error output 
fix https://github.com/chrislusf/seaweedfs/issues/2123
 2021-06-10 21:55:13 -07:00
Chris Lu
8b382a8209 refactor 2021-06-10 21:50:21 -07:00
Chris Lu
33b87244ef refactoring 2021-06-10 16:54:36 -07:00
Daniel Nagy
7ce3bee0c0
Check for history file errors as well 2021-06-08 11:14:49 +02:00
Daniel Nagy
0e3adde47f
Return correct exitcode when wheed upload fails 2021-06-07 23:38:14 +02:00
Chris Lu
76f24af79a 2.52 2021-06-07 12:13:23 -07:00
Chris Lu
5e6dfbc25f locks for data racing 2021-06-07 12:04:50 -07:00
Chris Lu
452c6ef183 limits concurrent uploads for one file 2021-06-06 23:05:17 -07:00
Chris Lu
8295e2feb6 skip md5 checking for now because of race condition 2021-06-06 22:16:32 -07:00
Chris Lu
c8dea3dd89 2.51 2021-06-06 21:54:00 -07:00
Chris Lu
19caeb7b02 fix writing the small file 2021-06-06 20:57:03 -07:00
Chris Lu
bb45dea15a filer: parallel data upload 2021-06-06 20:23:36 -07:00
Chris Lu
e00443a940 mount: adjust starting order 
avoid possible nil wfs.Server
 2021-06-06 20:22:42 -07:00
Chris Lu
44f1ba6894 refactor 2021-06-06 18:43:04 -07:00
Chris Lu
6c82326575 use bytes.Buffer to reduce memory allocation and gc 2021-06-06 13:42:36 -07:00
Chris Lu
9cba5cca0b optionally disable concurrent upload limit 2021-06-06 13:13:33 -07:00
Chris Lu
21ad9a4ac2 filer: mongodb avoids E11000 duplicate key error collection: seaweedfs.filemeta index: directory_1_name_1 dup key 2021-06-06 13:12:01 -07:00
Chris Lu
e6ba2f9c37 verbose but cleaner logic to invalidate fuse cache 2021-06-05 13:09:37 -07:00
Chris Lu
fadc1febdb FUSE: invalidate FUSE cached entries 
fix https://github.com/chrislusf/seaweedfs/issues/2108
 2021-06-05 02:23:07 -07:00
Chris Lu
ab606dec2a filer: add path-specific option to enforce readonly 2021-06-04 01:03:41 -07:00
Chris Lu
0a5388744c
Merge pull request #2105 from Woellchen/current_and_parent_dot_directories 
Return artificial . and .. directories
 2021-06-02 12:54:44 -07:00
Patrick Schmidt
77100754e6 Return artificial . and .. directories 2021-06-02 21:28:02 +02:00
Chris Lu
62142ff1d2
Merge pull request #2104 from kmlebedev/sync_volume_on_close 
avoid data loss after restarting a container with a volum server
 2021-06-02 11:47:57 -07:00
Chris Lu
cc34475012 remove file handle locking for setattr 2021-06-02 11:44:12 -07:00
Konstantin Lebedev
99ef280c7c avoid data loss after restarting a container with a volum server 2021-06-02 17:07:19 +05:00
Chris Lu
3db1642392 mount: skip persisting metadata if file is open 2021-06-02 03:40:08 -07:00
Chris Lu
556cc3a4ca mount: avoid exception if disk cache is not initialized 
related to https://github.com/chrislusf/seaweedfs/issues/2102
 2021-05-31 16:42:55 -07:00
Chris Lu
615cb24ba6 shell: use tab print style for completion 2021-05-31 03:29:29 -07:00
Chris Lu
c5de97ddbe keep alive for streaming connections 
fix https://github.com/chrislusf/seaweedfs/issues/2096

related to https://github.com/chrislusf/seaweedfs/issues/1277
 2021-05-31 01:26:24 -07:00
Chris Lu
2b60e2abb1 only disallow streaming signed when no auth enabled 
fix https://github.com/chrislusf/seaweedfs/issues/2101
 2021-05-31 01:03:04 -07:00
Chris Lu
4233ad3f07 2.50 2021-05-30 20:40:30 -07:00
Chris Lu
1456616a77 recreate grpc connections if too many errors 
address https://github.com/chrislusf/seaweedfs/issues/2098
 2021-05-30 00:07:43 -07:00
Chris Lu
fb8036385a s3: save metadata during put-object 
fix https://github.com/chrislusf/seaweedfs/issues/2092
 2021-05-29 14:14:30 -07:00
Chris Lu
2d7b4e5bb6 filer.backup: escape colon from path on windows 
fix https://github.com/chrislusf/seaweedfs/issues/2084
 2021-05-29 06:45:27 -07:00
Chris Lu
ae185b997f
Merge pull request #2099 from danielflira/mount-helper 
fix parameter multiple values
 2021-05-28 21:57:15 -07:00
danielflira
849f36c1ac fix parameter multiple values 2021-05-29 00:37:25 -03:00
Eugeniy Kozlov
6e8bd16819 freespace params fix 2021-05-28 14:19:24 +03:00
Chris Lu
4d55132c7d scaffold add comments 2021-05-27 14:40:25 -07:00
Chris Lu
45bffc92a8 filer.backup: fix cloud sinks when updating entry 2021-05-26 14:53:11 -07:00
Chris Lu
921e0d5008 remove verbose log 2021-05-26 14:43:34 -07:00
Chris Lu
1a70cb9b63 local sink: write and update files 
fix issues with https://github.com/chrislusf/seaweedfs/issues/2084
 2021-05-26 14:42:21 -07:00
Chris Lu
e699d16c85
Merge pull request #2085 from danielflira/mount-helper 
create fuse subcommand to use weed with mount
 2021-05-26 12:19:40 -07:00
danielflira
84488ebb33 replace filer.remote and parts[1] 2021-05-26 12:07:36 -03:00
Chris Lu
ac9bf71544 use jquery 3.6.0 2021-05-26 00:21:11 -07:00
danielflira
877c192af2 create fuse subcommand to use weed with mount 2021-05-25 23:32:35 -03:00
Chris Lu
67dd094b35 sqlite: does not support non-linux/darwin/windows 2021-05-25 17:36:15 -07:00
Chris Lu
dce1f02c9e filer.backup: backup to local directory optionally is incremental 
fixed one issue with https://github.com/chrislusf/seaweedfs/issues/2084
 2021-05-25 17:19:20 -07:00
Konstantin Lebedev
b612d5aebd s3 test get w/ If-Match: bogus ETag 2021-05-24 16:59:44 +05:00
Konstantin Lebedev
69c768870b - object write cache control 
- object write expires
 2021-05-24 15:43:55 +05:00
Konstantin Lebedev
84dce32a57
Merge branch 'master' into head_check_all_chunks 2021-05-24 12:28:19 +05:00
Konstantin Lebedev
0e404cd84b Chunk download stats 2021-05-24 12:14:50 +05:00
Konstantin Lebedev
bb42633522 rm chunk Download stats 2021-05-24 12:08:12 +05:00
Chris Lu
064269bb57 filer: Support sqlite as filer meta store 2021-05-23 23:58:28 -07:00
Chris Lu
b70aeb7585 bucket list only directories 2021-05-23 10:36:22 -07:00
Chris Lu
42fb03a66e 2.49 2021-05-23 00:51:47 -07:00
Chris Lu
3d3fa43542 filer: re-create grpc connections if having transport error 
fix https://github.com/chrislusf/seaweedfs/issues/2070
 2021-05-22 17:46:53 -07:00
Chris Lu
2d0a0733ec filer: bootstrap from peer filer 
fix https://github.com/chrislusf/seaweedfs/issues/1861
 2021-05-22 03:17:02 -07:00
Chris Lu
e5a2bf1287 s3: deprecating filer.options.buckets_fsync 2021-05-22 00:24:23 -07:00
Chris Lu
431684798b s3: add errors if requests are signed by no authentication is setup 
fix https://github.com/chrislusf/seaweedfs/issues/2075
 2021-05-21 14:08:47 -07:00
Chris Lu
2f136a04a1 organize a bit better for temp file directory 2021-05-21 13:02:18 -07:00
Konstantin Lebedev
3325b850cc rm func CheckAllChunkViews 2021-05-21 16:05:59 +05:00
Konstantin Lebedev
26a4f34a57 del checks 
url err logging
stats chunk fetch
 2021-05-21 15:59:12 +05:00
Chris Lu
e8b7d4ad64 simplify a bit 2021-05-21 02:10:44 -07:00
Chris Lu
0b00edfc3b waits in case master lost connection 2021-05-21 02:09:22 -07:00
Chris Lu
f37a4cbd0f randomize initial filer 2021-05-21 01:41:34 -07:00
Chris Lu
5d77840cff adjust help message 2021-05-21 01:38:57 -07:00
Chris Lu
dc1309f084 FUSE mount: support multiple filers 
fix https://github.com/chrislusf/seaweedfs/issues/2015
fix https://github.com/chrislusf/seaweedfs/issues/1531
 2021-05-21 01:28:00 -07:00
Konstantin Lebedev
42f631f549 fix https://github.com/chrislusf/seaweedfs/issues/2046 2021-05-20 13:19:35 +05:00
Chris Lu
30c67e3652 minor 2021-05-19 23:59:39 -07:00
Konstantin Lebedev
03d1199d5f Revert "revert PR #1903 avoid http error: superfluous response.WriteHeader" 
This reverts commit ac71117e
 2021-05-20 11:45:21 +05:00
Chris Lu
87a32bfef4 avoid possible nil when node is disconnected from its parent 
fix https://github.com/chrislusf/seaweedfs/issues/2073
 2021-05-19 10:02:01 -07:00
Chris Lu
1aa7e99a89 skip file not found error when deleting 2021-05-15 09:37:39 -07:00
Chris Lu
45a762223a 2.48 2021-05-14 10:26:42 -07:00
Chris Lu
789465d99e set renamed item to new directory 
related to https://github.com/chrislusf/seaweedfs/issues/2064
 2021-05-13 22:52:19 -07:00
Chris Lu
83c037e093 fix logs 2021-05-13 22:41:19 -07:00
Chris Lu
5a29d284fa ensure proper directory name 
a fix related to https://github.com/chrislusf/seaweedfs/issues/2064
 2021-05-13 10:18:35 -07:00
Chris Lu
3ff307e842 reduce logs 2021-05-13 00:32:37 -07:00
Chris Lu
a48ebd7c73 mount: read file when file is still being written 
a possible fix for https://github.com/chrislusf/seaweedfs/issues/2065
 2021-05-13 00:32:23 -07:00
Chris Lu
26a55bbb5c Adjust error message when bucket name conflicts with existing collections 
fix https://github.com/chrislusf/seaweedfs/issues/2069
 2021-05-12 22:30:39 -07:00
Chris Lu
ceb620a30a directory rename: change directory name after renaming 
fix https://github.com/chrislusf/seaweedfs/issues/2068
 2021-05-12 22:04:47 -07:00
Chris Lu
b430d1b6ee filer.copy: "check.size" before copying files 
fix https://github.com/chrislusf/seaweedfs/issues/2067
 2021-05-12 21:45:39 -07:00
Chris Lu
0f7b43af99 2.47 2021-05-11 10:12:15 -07:00
Chris Lu
d2d36a3f9d master: avoid creating too many volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2062
 2021-05-11 10:05:31 -07:00
Chris Lu
4596e64710 2.46 2021-05-10 21:58:37 -07:00
Chris Lu
e55aa41690 Merge branch 'upload_via_temp_file' 2021-05-10 21:48:18 -07:00
Chris Lu
1737af480a adjust logs 2021-05-10 21:47:51 -07:00
Chris Lu
d06ecc2649 working properly 2021-05-10 21:47:07 -07:00
Chris Lu
fe2edd4b50 2.45 2021-05-10 13:53:03 -07:00
Chris Lu
55d547afec switch back to in memory uploading for now 2021-05-10 13:50:10 -07:00
Chris Lu
9a6aa00e9d avoid nil locations 
fix https://github.com/chrislusf/seaweedfs/issues/2059
 2021-05-10 02:39:52 -07:00
Chris Lu
da0a4e775b Revert "Revert "Merge pull request #2027 from bingoohuang/master"" 
This reverts commit d74cdf0115.
 2021-05-10 00:13:55 -07:00
Chris Lu
aa6949ef25 2.44 2021-05-09 23:28:48 -07:00
Chris Lu
ba77833343 EOF handling 2021-05-09 23:04:24 -07:00
Chris Lu
f4cf30b497 mount: write to tmp file before uploading 2021-05-09 22:56:10 -07:00
Chris Lu
280e7cf289 mount: in case the set attribute is called before persisting the file 2021-05-09 22:55:30 -07:00
Chris Lu
74052064b6 add one missing optimization 2021-05-09 17:25:42 -07:00
Chris Lu
59ace54925 refactor 2021-05-09 17:22:30 -07:00
Chris Lu
50be19d23e refactor 2021-05-09 15:33:01 -07:00
Chris Lu
55e060cf61 refactor 2021-05-09 15:28:54 -07:00
Chris Lu
735e65be17 refactor 2021-05-09 15:22:38 -07:00
Chris Lu
93e84a12f2 refactor 2021-05-09 15:15:18 -07:00
Chris Lu
3942e3b2ef a better fix 2021-05-09 01:42:19 -07:00
Chris Lu
957e1a1bc1 fuse: important: if filer -filer.path is not root, directory listing will fail 2021-05-09 00:32:21 -07:00
Chris Lu
7ca75347ec minor 2021-05-07 21:56:45 -07:00
Chris Lu
8f8738867f add retry to assign volume 
fix https://github.com/chrislusf/seaweedfs/issues/2056
 2021-05-07 07:29:26 -07:00
Chris Lu
007401f3a0 remove duplicated code 2021-05-07 07:14:24 -07:00
Chris Lu
aaad4b578b
Merge pull request #2049 from qieqieplus/async-assign 
ahead of time volume assignment
 2021-05-06 10:00:25 -07:00
qieqieplus
c4d32f6937 ahead of time volume assignment 2021-05-06 18:55:44 +08:00
Chris Lu
55a8f57381 go fmt 2021-05-06 03:37:51 -07:00
Chris Lu
5753749c90 remove verbose logs 2021-05-06 03:34:34 -07:00
Chris Lu
38f411219a mount: skip local chunk cache if opened write only 2021-05-06 03:31:40 -07:00
Chris Lu
c899bdf063 a little optimization 2021-05-06 03:03:00 -07:00
Chris Lu
3eb336e0b0 report error only for the first multipart upload 
the glog.Errorf would always print for s3 multipart uploads
 2021-05-06 01:57:54 -07:00
Chris Lu
76c48ffe27 optional parallel copy ec shards 
fix https://github.com/chrislusf/seaweedfs/issues/2048
 2021-05-06 01:53:35 -07:00
Chris Lu
ac71117ee6 revert PR #1903 avoid http error: superfluous response.WriteHeader 2021-05-05 15:11:39 -07:00
Chris Lu
24efa31e49
Merge pull request #2045 from qieqieplus/fix-vacuum-commit 2021-05-05 07:41:38 -07:00
qieqieplus
ac26080bd2 fix concurrent vacuum & delete panic 2021-05-05 17:54:50 +08:00
Chris Lu
e24ba2aadc filer: delete specific tags 
fix https://github.com/chrislusf/seaweedfs/issues/2041
 2021-05-02 21:53:43 -07:00
Chris Lu
e87e6ef33c s3: return 404 if bucket does not exist 
fix https://github.com/chrislusf/seaweedfs/issues/2039
 2021-05-02 21:30:37 -07:00
Chris Lu
a4bb37a5fe add debug info 2021-05-01 16:10:26 -07:00
Chris Lu
2e56407c6b fix visited checking 2021-05-01 16:09:29 -07:00
Chris Lu
c48ef78670 2.43 2021-05-01 00:39:04 -07:00
Chris Lu
3a86d4dbfd mount: fix directory invalidation 
fix https://github.com/chrislusf/seaweedfs/issues/2038
 2021-04-30 22:51:06 -07:00
Chris Lu
d74cdf0115 Revert "Merge pull request #2027 from bingoohuang/master" 
Need to revert because docker image build failed. The docker apk package only has go 1.15.
 2021-04-30 03:36:15 -07:00
Chris Lu
84312e6799 2.42 2021-04-30 03:14:07 -07:00
Chris Lu
fa0d973113
Merge pull request #2037 from utsl42/reader_at 
make reader_at handle random reads more efficiently for FUSE
 2021-04-28 17:31:46 -07:00
Nathan Hawkins
042de9359c make reader_at handle random reads more efficiently for FUSE 2021-04-28 19:13:37 -04:00
Chris Lu
a26a37dfa3 fix compilation 
fix related to #2032
 2021-04-28 13:36:53 -07:00
Chris Lu
ef94ff6837
Merge pull request #2035 from kmlebedev/fix_chunks_etag 
fix aws style Etag for chunks
 2021-04-28 10:37:18 -07:00
Chris Lu
8ae3ea4b1d
Merge pull request #2034 from kmlebedev/iam_listkeys_filter 
iam ListAccessKeys filtred by username
 2021-04-28 10:34:40 -07:00
Chris Lu
9dca75aea8
Merge pull request #2032 from tobiasmuehl/patch-2 
Detect rar archives by mime type
 2021-04-28 10:31:19 -07:00
Konstantin Lebedev
c2269123d3 fix aws style Etag for chunks 2021-04-28 22:28:05 +05:00
Konstantin Lebedev
39f636c682 iam ListAccessKeys filtred by username 2021-04-28 17:15:22 +05:00
Tobias Mühl
a8864e2abd
Detect rar archives by mime type 
RAR archives might not have .rar extension, see [Wikipedia](https://en.wikipedia.org/wiki/RAR_(file_format))
 2021-04-28 15:54:19 +07:00
Tobias Mühl
12a7e87007
Do not compress brotli archives 2021-04-28 15:51:49 +07:00
Konstantin Lebedev
a48785c7df auth use bucket wild cards 2021-04-27 21:45:40 +05:00
bingoohuang
7a9d27fce8 promote to go:embed instead of github.com/rakyll/statik 2021-04-27 17:22:24 +08:00
bingoohuang
cf552417a7 minFreeSpace refactored 2021-04-27 10:37:24 +08:00
bingoohuang
31f1cdeac2 minFreeSpace argument allows size like 10GiB 2021-04-26 18:48:34 +08:00
Chris Lu
86185262bb 2.41 2021-04-24 16:54:36 -07:00
Chris Lu
79f2e780c1 ensure name pattern checking is case sensitive 2021-04-24 11:51:23 -07:00
Chris Lu
ddc8643ee0 filer: directory listing adds namePatternExclude 
fix https://github.com/chrislusf/seaweedfs/issues/2023
 2021-04-24 11:49:03 -07:00
Chris Lu
f0ad172e80 shell: show which server holds the lock 
fix https://github.com/chrislusf/seaweedfs/issues/1983
 2021-04-22 23:56:35 -07:00
Chris Lu
89eb9f6e70 clean up .uploads directory in mysql and postgres tables 
fix https://github.com/chrislusf/seaweedfs/issues/1957

When no uploads are running, you can run this SQL to clean up.
delete from <bucket_name> where directory like '/.uploads/%'
 2021-04-22 23:23:23 -07:00
Chris Lu
46ef1811a1 correct help message 2021-04-22 22:26:38 -07:00
Chris Lu
11c120c040 master UI adds volume size limit 2021-04-22 14:22:48 -07:00
Chris Lu
6e5df901e4 adjust package names 2021-04-22 14:22:48 -07:00
Chris Lu
ca998328c2 do not add new volumes when below minFreeSpacePercent 
fix https://github.com/chrislusf/seaweedfs/issues/2017
 2021-04-21 23:11:11 -07:00
Chris Lu
cd7bf1a72a filer.copy copy empty folders 
fix https://github.com/chrislusf/seaweedfs/issues/2016
 2021-04-21 11:17:43 -07:00
Chris Lu
a8114da02d avoid thundering herd effect 
transient errors may cause thundering herd effect to all trying to recover from remove ec shards
 2021-04-21 10:17:12 -07:00
Chris Lu
ae74d8f02a fix error message 
related to https://github.com/chrislusf/seaweedfs/issues/2012
 2021-04-21 01:40:16 -07:00
Chris Lu
11c405fc85 ensure file handles are released 2021-04-20 19:56:55 -07:00
liuxiaobo
c31c5e829c
fix path-specific filer store comment error 2021-04-20 10:08:58 +08:00
Chris Lu
83cf94ad2d delay new file creation unless file is opened exclusively 2021-04-19 10:58:25 -07:00
Chris Lu
e983f91b03 2.40 2021-04-18 13:58:01 -07:00
Chris Lu
d1c813c470 let the fuse library manage directory id 
otherwise, on mac, during large directory deletion, if some ReaDirAll happens, the lib seems confused about the directories, and some child directories are not deleted.
 2021-04-18 13:07:28 -07:00
Chris Lu
372872ebbf set root node inode number 2021-04-18 13:07:28 -07:00
Chris Lu
e332da4837 set inode value 2021-04-18 13:07:28 -07:00
Chris Lu
2acf6be24e resend the http request if connection is stale 2021-04-18 13:07:28 -07:00
Chris Lu
6cbd786db9 correctly runs git clone 2021-04-18 13:07:28 -07:00
Chris Lu
d41e6826d3 adjust logging 2021-04-18 13:06:38 -07:00
Chris Lu
d9a2a7f1c4 WIP 
no memory issue

if some directory is removed, it may have this error

$ rm -Rf ~/tmp/m2/s1
rm: fts_read: Device not configured
 2021-04-18 13:06:38 -07:00
Chris Lu
54410ca955 cleaner way to set readonly 2021-04-18 10:02:02 -07:00
Konstantin Lebedev
198688c717 revert volume etag 2021-04-16 23:22:31 +05:00
Chris Lu
c83ab91e2e remove unused variable 2021-04-16 10:34:02 -07:00
Chris Lu
3074e9b428 ensure consistent inode value 2021-04-15 22:42:24 -07:00
Chris Lu
b971317a16 avoid possible corrupted file names 2021-04-15 11:41:34 -07:00
Chris Lu
283d703d50 adjust text 2021-04-15 11:29:58 -07:00
Chris Lu
609e228578 avoid forward slash in file names 2021-04-15 10:53:04 -07:00
Chris Lu
ba92f2e714 add node.selectedVolumes 
fix https://github.com/chrislusf/seaweedfs/issues/1990
 2021-04-15 03:19:28 -07:00
Chris Lu
16c0304416 ensure to delete on filer also 2021-04-15 02:29:04 -07:00
Chris Lu
217e0f9066 mount: remove folder recursively 2021-04-15 01:51:10 -07:00
Chris Lu
3e669e6d7b mostly refactoring, add some error handling 2021-04-14 23:33:37 -07:00
Chris Lu
07f712c83f fix typo 2021-04-14 23:21:38 -07:00
Chris Lu
36c79de3f4 fuse mount: dir ReadDirAll avoid extra conversion to filer_pb.Entry 2021-04-14 23:21:24 -07:00
Chris Lu
e41766feb6 fuse mount: dir lookup avoids extra conversion to filer_pb.Entry object 2021-04-14 22:38:34 -07:00
Chris Lu
1adc8f86ea lighten up File object 
file.entry only exists when file.isOpen
 2021-04-14 20:49:15 -07:00
Chris Lu
6bc09b18c4 truncate is a bit faster to reuse the storage 2021-04-14 20:26:56 -07:00
Chris Lu
c04b7e106f mount: remove entry from Dir object 2021-04-14 20:26:13 -07:00
Chris Lu
fc0cbf565f add option to obfuscate the file names 2021-04-14 15:37:24 -07:00
Chris Lu
e75633c64f volume.check.disk: break loop for read only volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2002
 2021-04-14 12:40:13 -07:00
Chris Lu
9d50867d08 volume.tier.move: avoid data loss when destination volume server already has the volume 
fix https://github.com/chrislusf/seaweedfs/issues/2001
 2021-04-14 10:26:26 -07:00
Chris Lu
ff4c1d5965 adjust logging 
fix https://github.com/chrislusf/seaweedfs/issues/1999
 2021-04-14 10:04:26 -07:00
qieqieplus
270645f8d7 fix #1996 2021-04-14 18:29:28 +08:00
Chris Lu
90677e1097 ensure to call line.Close() 
fix https://github.com/chrislusf/seaweedfs/issues/1995

similar to https://github.com/peterh/liner/issues/104
 2021-04-14 01:33:21 -07:00
Chris Lu
3f3268cd1b go fmt 2021-04-14 00:30:16 -07:00
Chris Lu
ca0f07a188 move file reader, entryViewCache to file handle 
reduce file object size
 2021-04-14 00:29:58 -07:00
Chris Lu
5985a7d38d add log level possible values 
fix https://github.com/chrislusf/seaweedfs/issues/1989
 2021-04-12 21:15:51 -07:00
Chris Lu
f5de42fae3
Merge pull request #1975 from kmlebedev/iam_handlers 
IAM handlers
 2021-04-12 12:07:45 -07:00
Chris Lu
1e033d45b8 simpler logic 
related to https://github.com/chrislusf/seaweedfs/pull/1981
 2021-04-12 12:04:53 -07:00
Chris Lu
52200a903b
Merge pull request #1981 from ueni-ltd/if-modified-since 
Fix If-Modified-Since behavior
 2021-04-12 12:02:54 -07:00
Chris Lu
519b0e1e49 filer: upload to a directory without "/" suffix 
fix https://github.com/chrislusf/seaweedfs/issues/1988
 2021-04-12 11:56:56 -07:00
Konstantin Lebedev
8e02e138ea Merge branch 'upstreamMaster' into iam_handlers 2021-04-12 11:22:51 +05:00
Chris Lu
742ab1ec81 2.39 2021-04-11 19:47:11 -07:00
Chris Lu
0df5b53ad8 adjust help message 2021-04-11 00:26:28 -07:00
Chris Lu
f62c153274 go fmt 2021-04-10 23:48:18 -07:00
Chris Lu
af313dff58 add gateway for easier POST and DELETE blobs 2021-04-10 23:47:47 -07:00
Konstantin Lebedev
5021bea698 GetUserPolicy 2021-04-10 23:57:45 +05:00
Chris Lu
0b82edc0d2 filer: avoid stuck uploader 
fix https://github.com/chrislusf/seaweedfs/issues/1980

reverting the file upload batch executor
 2021-04-09 13:05:15 -07:00
Chris Lu
93f4146ffa properly release the view cache 2021-04-09 12:36:39 -07:00
Merlin Gaillard
f952f979d1 filer: return 304 when If-Modified-Since == Last-Modified 2021-04-09 15:04:17 +02:00
Merlin Gaillard
4d4acc715e s3api: handle 304 response code from filer 2021-04-09 12:13:19 +02:00
Chris Lu
6deb647a8f mount: fix possible memory leak 
if many files are read repeatedly, their metadata are accumulated in memory. This fix cleared the metadata after the file is read.
 2021-04-08 19:47:31 -07:00
Konstantin Lebedev
ba175f81b5 add auth aws signV4 2021-04-08 17:40:47 +05:00
Konstantin Lebedev
995ae91007 add DeleteUserPolicy 2021-04-08 11:16:36 +05:00
Philippe Pepiot
42a761ee20
Fix typo in weed filer long help 2021-04-07 22:47:23 +02:00
Chris Lu
3be061994f skip connection reset error 
fix https://github.com/chrislusf/seaweedfs/issues/1971

this is because the connections are pooled but the volume server has reset the connection
 2021-04-07 00:54:13 -07:00
Chris Lu
c5b08bac1b remove mac specific mount options 2021-04-07 00:54:13 -07:00
Konstantin Lebedev
f5f8eec8e2 fix get filerGrpcAddress 2021-04-06 13:53:56 +05:00
Konstantin Lebedev
011e6e90ee Merge branch 'upstreamMaster' into iamapipr 2021-04-06 13:50:33 +05:00
Konstantin Lebedev
ed79baa30f add tests 2021-04-06 13:43:08 +05:00
Chris Lu
2327c0756b fix to avoid loop 2021-04-05 23:24:26 -07:00
Chris Lu
a37eca78cd 2.38 2021-04-05 19:41:54 -07:00
Chris Lu
c131764c34 ensure error is sent back in channel 
fix https://github.com/chrislusf/seaweedfs/issues/1966

avoid shared readErr variable
 2021-04-05 19:40:12 -07:00
Chris Lu
0f64f5b9c8 mount: add readOnly option 
fix https://github.com/chrislusf/seaweedfs/issues/1961
 2021-04-04 21:40:58 -07:00
Chris Lu
6eee200c13 2.37 2021-04-04 18:45:48 -07:00
Chris Lu
fbb82a5c9c skip limiting if limit is zero 2021-04-04 18:38:33 -07:00
Chris Lu
8251d1140e refactor 2021-04-04 18:38:33 -07:00
Chris Lu
5dca98fe71 fix typo 2021-04-03 11:37:13 -07:00
Chris Lu
5e64f65632 ensure tables are created 
fix https://github.com/chrislusf/seaweedfs/issues/1957
 2021-04-02 12:02:26 -07:00
Chris Lu
319743d330 go fmt 2021-04-02 02:22:59 -07:00
Chris Lu
8e404a1433 go fmt 2021-04-02 02:22:26 -07:00
Chris Lu
bdf2ddddfd revert to same implementation as before 
This reverts commit 7e8edc3c4a.
 2021-04-02 02:21:38 -07:00
Chris Lu
67e019d54b add missing changes 2021-04-02 01:11:44 -07:00
Chris Lu
7e8edc3c4a refactoring 2021-04-02 01:10:24 -07:00
Chris Lu
cc0df36a9e smaller file chunks are faster 2021-04-01 02:21:59 -07:00
Chris Lu
cefe66f159 dedicated upload processor 
avoid thundering effect of overloading volume servers
 2021-04-01 02:21:40 -07:00
Chris Lu
1f984d2645 refactor buffer pool 2021-04-01 02:20:00 -07:00
Chris Lu
b5880334fc refactor 2021-03-30 21:07:34 -07:00
Chris Lu
e79e2ddeed remove unused MoveEvents 2021-03-30 20:42:44 -07:00
Chris Lu
e06807a55b adjust logs 2021-03-30 20:36:06 -07:00
Chris Lu
ac875976c0 filer, volume: add concurrent upload size limit to avoid OOM 
add some back pressure when writes are slow
 2021-03-30 02:10:53 -07:00
Chris Lu
a1e18a1384 server add memory profiling 2021-03-30 02:10:53 -07:00
LazyDBA247-Anyvision
9385e2224d fix postgres2 upsertQuery syntax also 2021-03-30 10:51:52 +03:00
Chris Lu
a5662c9e5e adjust ordering 2021-03-30 00:26:57 -07:00
Chris Lu
bec74c3e6a
Merge pull request #1948 from LazyDBA247-Anyvision/master 
add enableUpsert=true
 2021-03-30 00:25:01 -07:00
LazyDBA247-Anyvision
9f1cab179c use a compatible pg syntax for upsert 2021-03-30 10:21:27 +03:00
Chris Lu
a95929e53c reduce default concurrentWriters to 32 2021-03-30 00:17:52 -07:00
LazyDBA247-Anyvision
7f44d953b5 fix GetBool 2021-03-30 01:36:02 +03:00
LazyDBA247-Anyvision
4c51e6a660 add enableUpsert=true 
and rename config to upsertQuery
 2021-03-30 00:32:03 +03:00
Chris Lu
b61e9f6a1a adjust ordering 2021-03-29 00:49:50 -07:00
Chris Lu
1ebdbad222
Merge pull request #1946 from LazyDBA247-Anyvision/master 
Adding Customisation - insertQuery for postgres/mysql
 2021-03-29 00:42:48 -07:00
Konstantin Lebedev
8a95f9c10c iam GetUser 2021-03-29 12:01:44 +05:00
LazyDBA247-Anyvision
96c62bd34d scaffold - insertQuery 
Example for upsert query for mysql & postgres variants
 2021-03-29 10:01:12 +03:00
LazyDBA247-Anyvision
4a02389eb0 Adding custom insertQuery support for postgres/2 mysql/2 2021-03-29 09:58:13 +03:00
Chris Lu
6b7aa9633f 2.36 2021-03-28 19:09:06 -07:00
Chris Lu
bd7471d877 refactor 2021-03-25 12:05:59 -07:00
李海
06be5dc6c3 log snowflake sequencer nodeid's hex when start, it'll be in part of new assigned fid 2021-03-25 18:59:40 +08:00
李海
69b2dab9c6 add a snowflake sequencer as more robust fid generator, but less compressable than small auto-inc id 2021-03-25 18:49:26 +08:00
Konstantin Lebedev
40938d6a47 SaveInsideFiler S3 Configuration 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
9f26f2815c SaveAs S3 Configuration 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
82b0463fac handler PutUserPolicy 
https://docs.aws.amazon.com/IAM/latest/APIReference/API_PutUserPolicy.html
 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
d7719d0542 base handlers 2021-03-25 12:34:11 +05:00
Konstantin Lebedev
03c7953254 init Iam Api Server 2021-03-25 12:34:11 +05:00
Chris Lu
82dfe06066
Merge pull request #1933 from kmlebedev/missingKeysThreshold 
add missingKeysThreshold for fsck
 2021-03-24 10:56:25 -07:00
Konstantin Lebedev
020a5d40c3 avoid counter is null 2021-03-24 22:51:22 +05:00
Konstantin Lebedev
df6cf0a2fa nonRepairThreshold 2021-03-24 22:07:13 +05:00
Konstantin Lebedev
9ce82a9cd2 GOMAXPROCS use defaults to the value of runtime.NumCPU 
https://github.com/chrislusf/seaweedfs/issues/1942
 2021-03-24 18:45:58 +05:00
Konstantin Lebedev
60972f1c97 rename option repairThreshold 2021-03-24 13:24:49 +05:00