Chris Lu
ee6c67682c
minor
2021-06-12 02:52:41 -07:00
Chris Lu
9357911a95
remove all bucket metadata
...
fix https://github.com/chrislusf/seaweedfs/issues/2118
2021-06-10 23:37:54 -07:00
Chris Lu
b71c3cfba4
avoid possible empty object
2021-06-10 22:17:53 -07:00
Chris Lu
310e31424e
adjust the error output
...
fix https://github.com/chrislusf/seaweedfs/issues/2123
2021-06-10 21:55:13 -07:00
Chris Lu
8b382a8209
refactor
2021-06-10 21:50:21 -07:00
Chris Lu
33b87244ef
refactoring
2021-06-10 16:54:36 -07:00
Daniel Nagy
7ce3bee0c0
Check for history file errors as well
2021-06-08 11:14:49 +02:00
Daniel Nagy
0e3adde47f
Return correct exitcode when wheed upload
fails
2021-06-07 23:38:14 +02:00
Chris Lu
76f24af79a
2.52
2021-06-07 12:13:23 -07:00
Chris Lu
5e6dfbc25f
locks for data racing
2021-06-07 12:04:50 -07:00
Chris Lu
452c6ef183
limits concurrent uploads for one file
2021-06-06 23:05:17 -07:00
Chris Lu
8295e2feb6
skip md5 checking for now because of race condition
2021-06-06 22:16:32 -07:00
Chris Lu
c8dea3dd89
2.51
2021-06-06 21:54:00 -07:00
Chris Lu
19caeb7b02
fix writing the small file
2021-06-06 20:57:03 -07:00
Chris Lu
bb45dea15a
filer: parallel data upload
2021-06-06 20:23:36 -07:00
Chris Lu
e00443a940
mount: adjust starting order
...
avoid possible nil wfs.Server
2021-06-06 20:22:42 -07:00
Chris Lu
44f1ba6894
refactor
2021-06-06 18:43:04 -07:00
Chris Lu
6c82326575
use bytes.Buffer to reduce memory allocation and gc
2021-06-06 13:42:36 -07:00
Chris Lu
9cba5cca0b
optionally disable concurrent upload limit
2021-06-06 13:13:33 -07:00
Chris Lu
21ad9a4ac2
filer: mongodb avoids E11000 duplicate key error collection: seaweedfs.filemeta index: directory_1_name_1 dup key
2021-06-06 13:12:01 -07:00
Chris Lu
e6ba2f9c37
verbose but cleaner logic to invalidate fuse cache
2021-06-05 13:09:37 -07:00
Chris Lu
fadc1febdb
FUSE: invalidate FUSE cached entries
...
fix https://github.com/chrislusf/seaweedfs/issues/2108
2021-06-05 02:23:07 -07:00
Chris Lu
ab606dec2a
filer: add path-specific option to enforce readonly
2021-06-04 01:03:41 -07:00
Chris Lu
0a5388744c
Merge pull request #2105 from Woellchen/current_and_parent_dot_directories
...
Return artificial . and .. directories
2021-06-02 12:54:44 -07:00
Patrick Schmidt
77100754e6
Return artificial . and .. directories
2021-06-02 21:28:02 +02:00
Chris Lu
62142ff1d2
Merge pull request #2104 from kmlebedev/sync_volume_on_close
...
avoid data loss after restarting a container with a volum server
2021-06-02 11:47:57 -07:00
Chris Lu
cc34475012
remove file handle locking for setattr
2021-06-02 11:44:12 -07:00
Konstantin Lebedev
99ef280c7c
avoid data loss after restarting a container with a volum server
2021-06-02 17:07:19 +05:00
Chris Lu
3db1642392
mount: skip persisting metadata if file is open
2021-06-02 03:40:08 -07:00
Chris Lu
556cc3a4ca
mount: avoid exception if disk cache is not initialized
...
related to https://github.com/chrislusf/seaweedfs/issues/2102
2021-05-31 16:42:55 -07:00
Chris Lu
615cb24ba6
shell: use tab print style for completion
2021-05-31 03:29:29 -07:00
Chris Lu
c5de97ddbe
keep alive for streaming connections
...
fix https://github.com/chrislusf/seaweedfs/issues/2096
related to https://github.com/chrislusf/seaweedfs/issues/1277
2021-05-31 01:26:24 -07:00
Chris Lu
2b60e2abb1
only disallow streaming signed when no auth enabled
...
fix https://github.com/chrislusf/seaweedfs/issues/2101
2021-05-31 01:03:04 -07:00
Chris Lu
4233ad3f07
2.50
2021-05-30 20:40:30 -07:00
Chris Lu
1456616a77
recreate grpc connections if too many errors
...
address https://github.com/chrislusf/seaweedfs/issues/2098
2021-05-30 00:07:43 -07:00
Chris Lu
fb8036385a
s3: save metadata during put-object
...
fix https://github.com/chrislusf/seaweedfs/issues/2092
2021-05-29 14:14:30 -07:00
Chris Lu
2d7b4e5bb6
filer.backup: escape colon from path on windows
...
fix https://github.com/chrislusf/seaweedfs/issues/2084
2021-05-29 06:45:27 -07:00
Chris Lu
ae185b997f
Merge pull request #2099 from danielflira/mount-helper
...
fix parameter multiple values
2021-05-28 21:57:15 -07:00
danielflira
849f36c1ac
fix parameter multiple values
2021-05-29 00:37:25 -03:00
Eugeniy Kozlov
6e8bd16819
freespace params fix
2021-05-28 14:19:24 +03:00
Chris Lu
4d55132c7d
scaffold add comments
2021-05-27 14:40:25 -07:00
Chris Lu
45bffc92a8
filer.backup: fix cloud sinks when updating entry
2021-05-26 14:53:11 -07:00
Chris Lu
921e0d5008
remove verbose log
2021-05-26 14:43:34 -07:00
Chris Lu
1a70cb9b63
local sink: write and update files
...
fix issues with https://github.com/chrislusf/seaweedfs/issues/2084
2021-05-26 14:42:21 -07:00
Chris Lu
e699d16c85
Merge pull request #2085 from danielflira/mount-helper
...
create fuse subcommand to use weed with mount
2021-05-26 12:19:40 -07:00
danielflira
84488ebb33
replace filer.remote and parts[1]
2021-05-26 12:07:36 -03:00
Chris Lu
ac9bf71544
use jquery 3.6.0
2021-05-26 00:21:11 -07:00
danielflira
877c192af2
create fuse subcommand to use weed with mount
2021-05-25 23:32:35 -03:00
Chris Lu
67dd094b35
sqlite: does not support non-linux/darwin/windows
2021-05-25 17:36:15 -07:00
Chris Lu
dce1f02c9e
filer.backup: backup to local directory optionally is incremental
...
fixed one issue with https://github.com/chrislusf/seaweedfs/issues/2084
2021-05-25 17:19:20 -07:00
Konstantin Lebedev
b612d5aebd
s3 test get w/ If-Match: bogus ETag
2021-05-24 16:59:44 +05:00
Konstantin Lebedev
69c768870b
- object write cache control
...
- object write expires
2021-05-24 15:43:55 +05:00
Konstantin Lebedev
84dce32a57
Merge branch 'master' into head_check_all_chunks
2021-05-24 12:28:19 +05:00
Konstantin Lebedev
0e404cd84b
Chunk download stats
2021-05-24 12:14:50 +05:00
Konstantin Lebedev
bb42633522
rm chunk Download stats
2021-05-24 12:08:12 +05:00
Chris Lu
064269bb57
filer: Support sqlite as filer meta store
2021-05-23 23:58:28 -07:00
Chris Lu
b70aeb7585
bucket list only directories
2021-05-23 10:36:22 -07:00
Chris Lu
42fb03a66e
2.49
2021-05-23 00:51:47 -07:00
Chris Lu
3d3fa43542
filer: re-create grpc connections if having transport error
...
fix https://github.com/chrislusf/seaweedfs/issues/2070
2021-05-22 17:46:53 -07:00
Chris Lu
2d0a0733ec
filer: bootstrap from peer filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1861
2021-05-22 03:17:02 -07:00
Chris Lu
e5a2bf1287
s3: deprecating filer.options.buckets_fsync
2021-05-22 00:24:23 -07:00
Chris Lu
431684798b
s3: add errors if requests are signed by no authentication is setup
...
fix https://github.com/chrislusf/seaweedfs/issues/2075
2021-05-21 14:08:47 -07:00
Chris Lu
2f136a04a1
organize a bit better for temp file directory
2021-05-21 13:02:18 -07:00
Konstantin Lebedev
3325b850cc
rm func CheckAllChunkViews
2021-05-21 16:05:59 +05:00
Konstantin Lebedev
26a4f34a57
del checks
...
url err logging
stats chunk fetch
2021-05-21 15:59:12 +05:00
Chris Lu
e8b7d4ad64
simplify a bit
2021-05-21 02:10:44 -07:00
Chris Lu
0b00edfc3b
waits in case master lost connection
2021-05-21 02:09:22 -07:00
Chris Lu
f37a4cbd0f
randomize initial filer
2021-05-21 01:41:34 -07:00
Chris Lu
5d77840cff
adjust help message
2021-05-21 01:38:57 -07:00
Chris Lu
dc1309f084
FUSE mount: support multiple filers
...
fix https://github.com/chrislusf/seaweedfs/issues/2015
fix https://github.com/chrislusf/seaweedfs/issues/1531
2021-05-21 01:28:00 -07:00
Konstantin Lebedev
42f631f549
fix https://github.com/chrislusf/seaweedfs/issues/2046
2021-05-20 13:19:35 +05:00
Chris Lu
30c67e3652
minor
2021-05-19 23:59:39 -07:00
Konstantin Lebedev
03d1199d5f
Revert "revert PR #1903 avoid http error: superfluous response.WriteHeader"
...
This reverts commit ac71117e
2021-05-20 11:45:21 +05:00
Chris Lu
87a32bfef4
avoid possible nil when node is disconnected from its parent
...
fix https://github.com/chrislusf/seaweedfs/issues/2073
2021-05-19 10:02:01 -07:00
Chris Lu
1aa7e99a89
skip file not found error when deleting
2021-05-15 09:37:39 -07:00
Chris Lu
45a762223a
2.48
2021-05-14 10:26:42 -07:00
Chris Lu
789465d99e
set renamed item to new directory
...
related to https://github.com/chrislusf/seaweedfs/issues/2064
2021-05-13 22:52:19 -07:00
Chris Lu
83c037e093
fix logs
2021-05-13 22:41:19 -07:00
Chris Lu
5a29d284fa
ensure proper directory name
...
a fix related to https://github.com/chrislusf/seaweedfs/issues/2064
2021-05-13 10:18:35 -07:00
Chris Lu
3ff307e842
reduce logs
2021-05-13 00:32:37 -07:00
Chris Lu
a48ebd7c73
mount: read file when file is still being written
...
a possible fix for https://github.com/chrislusf/seaweedfs/issues/2065
2021-05-13 00:32:23 -07:00
Chris Lu
26a55bbb5c
Adjust error message when bucket name conflicts with existing collections
...
fix https://github.com/chrislusf/seaweedfs/issues/2069
2021-05-12 22:30:39 -07:00
Chris Lu
ceb620a30a
directory rename: change directory name after renaming
...
fix https://github.com/chrislusf/seaweedfs/issues/2068
2021-05-12 22:04:47 -07:00
Chris Lu
b430d1b6ee
filer.copy: "check.size" before copying files
...
fix https://github.com/chrislusf/seaweedfs/issues/2067
2021-05-12 21:45:39 -07:00
Chris Lu
0f7b43af99
2.47
2021-05-11 10:12:15 -07:00
Chris Lu
d2d36a3f9d
master: avoid creating too many volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/2062
2021-05-11 10:05:31 -07:00
Chris Lu
4596e64710
2.46
2021-05-10 21:58:37 -07:00
Chris Lu
e55aa41690
Merge branch 'upload_via_temp_file'
2021-05-10 21:48:18 -07:00
Chris Lu
1737af480a
adjust logs
2021-05-10 21:47:51 -07:00
Chris Lu
d06ecc2649
working properly
2021-05-10 21:47:07 -07:00
Chris Lu
fe2edd4b50
2.45
2021-05-10 13:53:03 -07:00
Chris Lu
55d547afec
switch back to in memory uploading for now
2021-05-10 13:50:10 -07:00
Chris Lu
9a6aa00e9d
avoid nil locations
...
fix https://github.com/chrislusf/seaweedfs/issues/2059
2021-05-10 02:39:52 -07:00
Chris Lu
da0a4e775b
Revert "Revert "Merge pull request #2027 from bingoohuang/master""
...
This reverts commit d74cdf0115
.
2021-05-10 00:13:55 -07:00
Chris Lu
aa6949ef25
2.44
2021-05-09 23:28:48 -07:00
Chris Lu
ba77833343
EOF handling
2021-05-09 23:04:24 -07:00
Chris Lu
f4cf30b497
mount: write to tmp file before uploading
2021-05-09 22:56:10 -07:00
Chris Lu
280e7cf289
mount: in case the set attribute is called before persisting the file
2021-05-09 22:55:30 -07:00
Chris Lu
74052064b6
add one missing optimization
2021-05-09 17:25:42 -07:00
Chris Lu
59ace54925
refactor
2021-05-09 17:22:30 -07:00
Chris Lu
50be19d23e
refactor
2021-05-09 15:33:01 -07:00
Chris Lu
55e060cf61
refactor
2021-05-09 15:28:54 -07:00
Chris Lu
735e65be17
refactor
2021-05-09 15:22:38 -07:00
Chris Lu
93e84a12f2
refactor
2021-05-09 15:15:18 -07:00
Chris Lu
3942e3b2ef
a better fix
2021-05-09 01:42:19 -07:00
Chris Lu
957e1a1bc1
fuse: important: if filer -filer.path is not root, directory listing will fail
2021-05-09 00:32:21 -07:00
Chris Lu
7ca75347ec
minor
2021-05-07 21:56:45 -07:00
Chris Lu
8f8738867f
add retry to assign volume
...
fix https://github.com/chrislusf/seaweedfs/issues/2056
2021-05-07 07:29:26 -07:00
Chris Lu
007401f3a0
remove duplicated code
2021-05-07 07:14:24 -07:00
Chris Lu
aaad4b578b
Merge pull request #2049 from qieqieplus/async-assign
...
ahead of time volume assignment
2021-05-06 10:00:25 -07:00
qieqieplus
c4d32f6937
ahead of time volume assignment
2021-05-06 18:55:44 +08:00
Chris Lu
55a8f57381
go fmt
2021-05-06 03:37:51 -07:00
Chris Lu
5753749c90
remove verbose logs
2021-05-06 03:34:34 -07:00
Chris Lu
38f411219a
mount: skip local chunk cache if opened write only
2021-05-06 03:31:40 -07:00
Chris Lu
c899bdf063
a little optimization
2021-05-06 03:03:00 -07:00
Chris Lu
3eb336e0b0
report error only for the first multipart upload
...
the glog.Errorf would always print for s3 multipart uploads
2021-05-06 01:57:54 -07:00
Chris Lu
76c48ffe27
optional parallel copy ec shards
...
fix https://github.com/chrislusf/seaweedfs/issues/2048
2021-05-06 01:53:35 -07:00
Chris Lu
ac71117ee6
revert PR #1903 avoid http error: superfluous response.WriteHeader
2021-05-05 15:11:39 -07:00
Chris Lu
24efa31e49
Merge pull request #2045 from qieqieplus/fix-vacuum-commit
2021-05-05 07:41:38 -07:00
qieqieplus
ac26080bd2
fix concurrent vacuum & delete panic
2021-05-05 17:54:50 +08:00
Chris Lu
e24ba2aadc
filer: delete specific tags
...
fix https://github.com/chrislusf/seaweedfs/issues/2041
2021-05-02 21:53:43 -07:00
Chris Lu
e87e6ef33c
s3: return 404 if bucket does not exist
...
fix https://github.com/chrislusf/seaweedfs/issues/2039
2021-05-02 21:30:37 -07:00
Chris Lu
a4bb37a5fe
add debug info
2021-05-01 16:10:26 -07:00
Chris Lu
2e56407c6b
fix visited checking
2021-05-01 16:09:29 -07:00
Chris Lu
c48ef78670
2.43
2021-05-01 00:39:04 -07:00
Chris Lu
3a86d4dbfd
mount: fix directory invalidation
...
fix https://github.com/chrislusf/seaweedfs/issues/2038
2021-04-30 22:51:06 -07:00
Chris Lu
d74cdf0115
Revert "Merge pull request #2027 from bingoohuang/master"
...
Need to revert because docker image build failed. The docker apk package only has go 1.15.
2021-04-30 03:36:15 -07:00
Chris Lu
84312e6799
2.42
2021-04-30 03:14:07 -07:00
Chris Lu
fa0d973113
Merge pull request #2037 from utsl42/reader_at
...
make reader_at handle random reads more efficiently for FUSE
2021-04-28 17:31:46 -07:00
Nathan Hawkins
042de9359c
make reader_at handle random reads more efficiently for FUSE
2021-04-28 19:13:37 -04:00
Chris Lu
a26a37dfa3
fix compilation
...
fix related to #2032
2021-04-28 13:36:53 -07:00
Chris Lu
ef94ff6837
Merge pull request #2035 from kmlebedev/fix_chunks_etag
...
fix aws style Etag for chunks
2021-04-28 10:37:18 -07:00
Chris Lu
8ae3ea4b1d
Merge pull request #2034 from kmlebedev/iam_listkeys_filter
...
iam ListAccessKeys filtred by username
2021-04-28 10:34:40 -07:00
Chris Lu
9dca75aea8
Merge pull request #2032 from tobiasmuehl/patch-2
...
Detect rar archives by mime type
2021-04-28 10:31:19 -07:00
Konstantin Lebedev
c2269123d3
fix aws style Etag for chunks
2021-04-28 22:28:05 +05:00
Konstantin Lebedev
39f636c682
iam ListAccessKeys filtred by username
2021-04-28 17:15:22 +05:00
Tobias Mühl
a8864e2abd
Detect rar archives by mime type
...
RAR archives might not have .rar extension, see [Wikipedia](https://en.wikipedia.org/wiki/RAR_(file_format) )
2021-04-28 15:54:19 +07:00
Tobias Mühl
12a7e87007
Do not compress brotli archives
2021-04-28 15:51:49 +07:00
Konstantin Lebedev
a48785c7df
auth use bucket wild cards
2021-04-27 21:45:40 +05:00
bingoohuang
7a9d27fce8
promote to go:embed instead of github.com/rakyll/statik
2021-04-27 17:22:24 +08:00
bingoohuang
cf552417a7
minFreeSpace refactored
2021-04-27 10:37:24 +08:00
bingoohuang
31f1cdeac2
minFreeSpace argument allows size like 10GiB
2021-04-26 18:48:34 +08:00
Chris Lu
86185262bb
2.41
2021-04-24 16:54:36 -07:00
Chris Lu
79f2e780c1
ensure name pattern checking is case sensitive
2021-04-24 11:51:23 -07:00
Chris Lu
ddc8643ee0
filer: directory listing adds namePatternExclude
...
fix https://github.com/chrislusf/seaweedfs/issues/2023
2021-04-24 11:49:03 -07:00
Chris Lu
f0ad172e80
shell: show which server holds the lock
...
fix https://github.com/chrislusf/seaweedfs/issues/1983
2021-04-22 23:56:35 -07:00
Chris Lu
89eb9f6e70
clean up .uploads directory in mysql and postgres tables
...
fix https://github.com/chrislusf/seaweedfs/issues/1957
When no uploads are running, you can run this SQL to clean up.
delete from <bucket_name> where directory like '/.uploads/%'
2021-04-22 23:23:23 -07:00
Chris Lu
46ef1811a1
correct help message
2021-04-22 22:26:38 -07:00
Chris Lu
11c120c040
master UI adds volume size limit
2021-04-22 14:22:48 -07:00
Chris Lu
6e5df901e4
adjust package names
2021-04-22 14:22:48 -07:00
Chris Lu
ca998328c2
do not add new volumes when below minFreeSpacePercent
...
fix https://github.com/chrislusf/seaweedfs/issues/2017
2021-04-21 23:11:11 -07:00
Chris Lu
cd7bf1a72a
filer.copy copy empty folders
...
fix https://github.com/chrislusf/seaweedfs/issues/2016
2021-04-21 11:17:43 -07:00
Chris Lu
a8114da02d
avoid thundering herd effect
...
transient errors may cause thundering herd effect to all trying to recover from remove ec shards
2021-04-21 10:17:12 -07:00
Chris Lu
ae74d8f02a
fix error message
...
related to https://github.com/chrislusf/seaweedfs/issues/2012
2021-04-21 01:40:16 -07:00
Chris Lu
11c405fc85
ensure file handles are released
2021-04-20 19:56:55 -07:00
liuxiaobo
c31c5e829c
fix path-specific filer store comment error
2021-04-20 10:08:58 +08:00
Chris Lu
83cf94ad2d
delay new file creation unless file is opened exclusively
2021-04-19 10:58:25 -07:00
Chris Lu
e983f91b03
2.40
2021-04-18 13:58:01 -07:00
Chris Lu
d1c813c470
let the fuse library manage directory id
...
otherwise, on mac, during large directory deletion, if some ReaDirAll happens, the lib seems confused about the directories, and some child directories are not deleted.
2021-04-18 13:07:28 -07:00
Chris Lu
372872ebbf
set root node inode number
2021-04-18 13:07:28 -07:00
Chris Lu
e332da4837
set inode value
2021-04-18 13:07:28 -07:00
Chris Lu
2acf6be24e
resend the http request if connection is stale
2021-04-18 13:07:28 -07:00
Chris Lu
6cbd786db9
correctly runs git clone
2021-04-18 13:07:28 -07:00
Chris Lu
d41e6826d3
adjust logging
2021-04-18 13:06:38 -07:00
Chris Lu
d9a2a7f1c4
WIP
...
no memory issue
if some directory is removed, it may have this error
$ rm -Rf ~/tmp/m2/s1
rm: fts_read: Device not configured
2021-04-18 13:06:38 -07:00
Chris Lu
54410ca955
cleaner way to set readonly
2021-04-18 10:02:02 -07:00
Konstantin Lebedev
198688c717
revert volume etag
2021-04-16 23:22:31 +05:00
Chris Lu
c83ab91e2e
remove unused variable
2021-04-16 10:34:02 -07:00
Chris Lu
3074e9b428
ensure consistent inode value
2021-04-15 22:42:24 -07:00
Chris Lu
b971317a16
avoid possible corrupted file names
2021-04-15 11:41:34 -07:00
Chris Lu
283d703d50
adjust text
2021-04-15 11:29:58 -07:00
Chris Lu
609e228578
avoid forward slash in file names
2021-04-15 10:53:04 -07:00
Chris Lu
ba92f2e714
add node.selectedVolumes
...
fix https://github.com/chrislusf/seaweedfs/issues/1990
2021-04-15 03:19:28 -07:00
Chris Lu
16c0304416
ensure to delete on filer also
2021-04-15 02:29:04 -07:00
Chris Lu
217e0f9066
mount: remove folder recursively
2021-04-15 01:51:10 -07:00
Chris Lu
3e669e6d7b
mostly refactoring, add some error handling
2021-04-14 23:33:37 -07:00
Chris Lu
07f712c83f
fix typo
2021-04-14 23:21:38 -07:00
Chris Lu
36c79de3f4
fuse mount: dir ReadDirAll avoid extra conversion to filer_pb.Entry
2021-04-14 23:21:24 -07:00
Chris Lu
e41766feb6
fuse mount: dir lookup avoids extra conversion to filer_pb.Entry object
2021-04-14 22:38:34 -07:00
Chris Lu
1adc8f86ea
lighten up File object
...
file.entry only exists when file.isOpen
2021-04-14 20:49:15 -07:00
Chris Lu
6bc09b18c4
truncate is a bit faster to reuse the storage
2021-04-14 20:26:56 -07:00
Chris Lu
c04b7e106f
mount: remove entry from Dir object
2021-04-14 20:26:13 -07:00
Chris Lu
fc0cbf565f
add option to obfuscate the file names
2021-04-14 15:37:24 -07:00
Chris Lu
e75633c64f
volume.check.disk: break loop for read only volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/2002
2021-04-14 12:40:13 -07:00
Chris Lu
9d50867d08
volume.tier.move: avoid data loss when destination volume server already has the volume
...
fix https://github.com/chrislusf/seaweedfs/issues/2001
2021-04-14 10:26:26 -07:00
Chris Lu
ff4c1d5965
adjust logging
...
fix https://github.com/chrislusf/seaweedfs/issues/1999
2021-04-14 10:04:26 -07:00
qieqieplus
270645f8d7
fix #1996
2021-04-14 18:29:28 +08:00
Chris Lu
90677e1097
ensure to call line.Close()
...
fix https://github.com/chrislusf/seaweedfs/issues/1995
similar to https://github.com/peterh/liner/issues/104
2021-04-14 01:33:21 -07:00
Chris Lu
3f3268cd1b
go fmt
2021-04-14 00:30:16 -07:00
Chris Lu
ca0f07a188
move file reader, entryViewCache to file handle
...
reduce file object size
2021-04-14 00:29:58 -07:00
Chris Lu
5985a7d38d
add log level possible values
...
fix https://github.com/chrislusf/seaweedfs/issues/1989
2021-04-12 21:15:51 -07:00
Chris Lu
f5de42fae3
Merge pull request #1975 from kmlebedev/iam_handlers
...
IAM handlers
2021-04-12 12:07:45 -07:00
Chris Lu
1e033d45b8
simpler logic
...
related to https://github.com/chrislusf/seaweedfs/pull/1981
2021-04-12 12:04:53 -07:00
Chris Lu
52200a903b
Merge pull request #1981 from ueni-ltd/if-modified-since
...
Fix If-Modified-Since behavior
2021-04-12 12:02:54 -07:00
Chris Lu
519b0e1e49
filer: upload to a directory without "/" suffix
...
fix https://github.com/chrislusf/seaweedfs/issues/1988
2021-04-12 11:56:56 -07:00
Konstantin Lebedev
8e02e138ea
Merge branch 'upstreamMaster' into iam_handlers
2021-04-12 11:22:51 +05:00
Chris Lu
742ab1ec81
2.39
2021-04-11 19:47:11 -07:00
Chris Lu
0df5b53ad8
adjust help message
2021-04-11 00:26:28 -07:00
Chris Lu
f62c153274
go fmt
2021-04-10 23:48:18 -07:00
Chris Lu
af313dff58
add gateway for easier POST and DELETE blobs
2021-04-10 23:47:47 -07:00
Konstantin Lebedev
5021bea698
GetUserPolicy
2021-04-10 23:57:45 +05:00
Chris Lu
0b82edc0d2
filer: avoid stuck uploader
...
fix https://github.com/chrislusf/seaweedfs/issues/1980
reverting the file upload batch executor
2021-04-09 13:05:15 -07:00
Chris Lu
93f4146ffa
properly release the view cache
2021-04-09 12:36:39 -07:00
Merlin Gaillard
f952f979d1
filer: return 304 when If-Modified-Since == Last-Modified
2021-04-09 15:04:17 +02:00
Merlin Gaillard
4d4acc715e
s3api: handle 304 response code from filer
2021-04-09 12:13:19 +02:00
Chris Lu
6deb647a8f
mount: fix possible memory leak
...
if many files are read repeatedly, their metadata are accumulated in memory. This fix cleared the metadata after the file is read.
2021-04-08 19:47:31 -07:00
Konstantin Lebedev
ba175f81b5
add auth aws signV4
2021-04-08 17:40:47 +05:00
Konstantin Lebedev
995ae91007
add DeleteUserPolicy
2021-04-08 11:16:36 +05:00
Philippe Pepiot
42a761ee20
Fix typo in weed filer long help
2021-04-07 22:47:23 +02:00
Chris Lu
3be061994f
skip connection reset error
...
fix https://github.com/chrislusf/seaweedfs/issues/1971
this is because the connections are pooled but the volume server has reset the connection
2021-04-07 00:54:13 -07:00
Chris Lu
c5b08bac1b
remove mac specific mount options
2021-04-07 00:54:13 -07:00
Konstantin Lebedev
f5f8eec8e2
fix get filerGrpcAddress
2021-04-06 13:53:56 +05:00
Konstantin Lebedev
011e6e90ee
Merge branch 'upstreamMaster' into iamapipr
2021-04-06 13:50:33 +05:00
Konstantin Lebedev
ed79baa30f
add tests
2021-04-06 13:43:08 +05:00
Chris Lu
2327c0756b
fix to avoid loop
2021-04-05 23:24:26 -07:00
Chris Lu
a37eca78cd
2.38
2021-04-05 19:41:54 -07:00
Chris Lu
c131764c34
ensure error is sent back in channel
...
fix https://github.com/chrislusf/seaweedfs/issues/1966
avoid shared readErr variable
2021-04-05 19:40:12 -07:00
Chris Lu
0f64f5b9c8
mount: add readOnly option
...
fix https://github.com/chrislusf/seaweedfs/issues/1961
2021-04-04 21:40:58 -07:00
Chris Lu
6eee200c13
2.37
2021-04-04 18:45:48 -07:00
Chris Lu
fbb82a5c9c
skip limiting if limit is zero
2021-04-04 18:38:33 -07:00
Chris Lu
8251d1140e
refactor
2021-04-04 18:38:33 -07:00
Chris Lu
5dca98fe71
fix typo
2021-04-03 11:37:13 -07:00
Chris Lu
5e64f65632
ensure tables are created
...
fix https://github.com/chrislusf/seaweedfs/issues/1957
2021-04-02 12:02:26 -07:00
Chris Lu
319743d330
go fmt
2021-04-02 02:22:59 -07:00
Chris Lu
8e404a1433
go fmt
2021-04-02 02:22:26 -07:00
Chris Lu
bdf2ddddfd
revert to same implementation as before
...
This reverts commit 7e8edc3c4a
.
2021-04-02 02:21:38 -07:00
Chris Lu
67e019d54b
add missing changes
2021-04-02 01:11:44 -07:00
Chris Lu
7e8edc3c4a
refactoring
2021-04-02 01:10:24 -07:00
Chris Lu
cc0df36a9e
smaller file chunks are faster
2021-04-01 02:21:59 -07:00
Chris Lu
cefe66f159
dedicated upload processor
...
avoid thundering effect of overloading volume servers
2021-04-01 02:21:40 -07:00
Chris Lu
1f984d2645
refactor buffer pool
2021-04-01 02:20:00 -07:00
Chris Lu
b5880334fc
refactor
2021-03-30 21:07:34 -07:00
Chris Lu
e79e2ddeed
remove unused MoveEvents
2021-03-30 20:42:44 -07:00
Chris Lu
e06807a55b
adjust logs
2021-03-30 20:36:06 -07:00
Chris Lu
ac875976c0
filer, volume: add concurrent upload size limit to avoid OOM
...
add some back pressure when writes are slow
2021-03-30 02:10:53 -07:00
Chris Lu
a1e18a1384
server add memory profiling
2021-03-30 02:10:53 -07:00
LazyDBA247-Anyvision
9385e2224d
fix postgres2 upsertQuery syntax also
2021-03-30 10:51:52 +03:00
Chris Lu
a5662c9e5e
adjust ordering
2021-03-30 00:26:57 -07:00
Chris Lu
bec74c3e6a
Merge pull request #1948 from LazyDBA247-Anyvision/master
...
add enableUpsert=true
2021-03-30 00:25:01 -07:00
LazyDBA247-Anyvision
9f1cab179c
use a compatible pg syntax for upsert
2021-03-30 10:21:27 +03:00
Chris Lu
a95929e53c
reduce default concurrentWriters to 32
2021-03-30 00:17:52 -07:00
LazyDBA247-Anyvision
7f44d953b5
fix GetBool
2021-03-30 01:36:02 +03:00
LazyDBA247-Anyvision
4c51e6a660
add enableUpsert=true
...
and rename config to upsertQuery
2021-03-30 00:32:03 +03:00
Chris Lu
b61e9f6a1a
adjust ordering
2021-03-29 00:49:50 -07:00
Chris Lu
1ebdbad222
Merge pull request #1946 from LazyDBA247-Anyvision/master
...
Adding Customisation - insertQuery for postgres/mysql
2021-03-29 00:42:48 -07:00
Konstantin Lebedev
8a95f9c10c
iam GetUser
2021-03-29 12:01:44 +05:00
LazyDBA247-Anyvision
96c62bd34d
scaffold - insertQuery
...
Example for upsert query for mysql & postgres variants
2021-03-29 10:01:12 +03:00
LazyDBA247-Anyvision
4a02389eb0
Adding custom insertQuery support for postgres/2 mysql/2
2021-03-29 09:58:13 +03:00
Chris Lu
6b7aa9633f
2.36
2021-03-28 19:09:06 -07:00
Chris Lu
bd7471d877
refactor
2021-03-25 12:05:59 -07:00
李海
06be5dc6c3
log snowflake sequencer nodeid's hex when start, it'll be in part of new assigned fid
2021-03-25 18:59:40 +08:00
李海
69b2dab9c6
add a snowflake sequencer as more robust fid generator, but less compressable than small auto-inc id
2021-03-25 18:49:26 +08:00
Konstantin Lebedev
40938d6a47
SaveInsideFiler S3 Configuration
2021-03-25 12:34:11 +05:00
Konstantin Lebedev
9f26f2815c
SaveAs S3 Configuration
2021-03-25 12:34:11 +05:00
Konstantin Lebedev
82b0463fac
handler PutUserPolicy
...
https://docs.aws.amazon.com/IAM/latest/APIReference/API_PutUserPolicy.html
2021-03-25 12:34:11 +05:00
Konstantin Lebedev
d7719d0542
base handlers
2021-03-25 12:34:11 +05:00
Konstantin Lebedev
03c7953254
init Iam Api Server
2021-03-25 12:34:11 +05:00
Chris Lu
82dfe06066
Merge pull request #1933 from kmlebedev/missingKeysThreshold
...
add missingKeysThreshold for fsck
2021-03-24 10:56:25 -07:00
Konstantin Lebedev
020a5d40c3
avoid counter is null
2021-03-24 22:51:22 +05:00
Konstantin Lebedev
df6cf0a2fa
nonRepairThreshold
2021-03-24 22:07:13 +05:00
Konstantin Lebedev
9ce82a9cd2
GOMAXPROCS use defaults to the value of runtime.NumCPU
...
https://github.com/chrislusf/seaweedfs/issues/1942
2021-03-24 18:45:58 +05:00
Konstantin Lebedev
60972f1c97
rename option repairThreshold
2021-03-24 13:24:49 +05:00
Chris Lu
a801332b0d
filer: return 409 if file conflicts with a directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1938
2021-03-24 01:03:11 -07:00
Chris Lu
c42b95c596
more help message
...
fix https://github.com/chrislusf/seaweedfs/issues/1937
2021-03-23 17:27:57 -07:00
Konstantin Lebedev
40a9e88c07
rename var
2021-03-23 21:42:35 +05:00
Konstantin Lebedev
d21a5bf5d0
add missingKeysThreshold for fsck
2021-03-23 15:04:07 +05:00
Chris Lu
85cff10787
mysql: ensure name column is case sensitive
...
fix https://github.com/chrislusf/seaweedfs/issues/1931
2021-03-23 00:46:50 -07:00
Chris Lu
4abb511db3
make a local copy of the in memory cached data
2021-03-22 22:33:07 -07:00
Chris Lu
3cbc40fa48
avoid creating multiple reader
2021-03-22 22:32:47 -07:00
Chris Lu
2f7c7afdec
refactor: remove unused parameter
2021-03-22 22:13:19 -07:00
Chris Lu
d48dd0c738
use []byte directly instead of bytes.Buffer
2021-03-22 22:12:57 -07:00
Chris Lu
8eb14967ef
shell: volume.check.disk adds optional slow mode
...
fix https://github.com/chrislusf/seaweedfs/issues/1930
2021-03-22 21:01:43 -07:00
Chris Lu
288369cfc7
mount: release resources when Forget() is called
...
address https://github.com/chrislusf/seaweedfs/issues/1929
2021-03-22 16:22:19 -07:00
Chris Lu
5d931eff27
avoid possible nil
...
fix https://github.com/chrislusf/seaweedfs/issues/1928
The nil was because of `dn.Parent().UnlinkChildNode(dn.Id())` in topo.UnRegisterDataNode() function, when the dn leaves the cluster.
2021-03-22 13:24:07 -07:00
Chris Lu
f315eb2bb8
fix build error
2021-03-22 00:12:53 -07:00
Chris Lu
1dd5bc134c
2.35
2021-03-22 00:05:09 -07:00
Chris Lu
b465095db1
shell: add volume.check.disk to fix inconsistency for replicated volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/1923
2021-03-22 00:03:16 -07:00
Chris Lu
df461402cc
ensure entry attribute is not nill
...
fix https://github.com/chrislusf/seaweedfs/issues/1926
2021-03-21 22:27:30 -07:00
Chris Lu
102a951377
refactor, split into 2 files
2021-03-21 13:05:53 -07:00
Chris Lu
f6e2566ee5
better logs for cookie mismatch
2021-03-20 12:02:57 -07:00
Chris Lu
27e24162f2
minor
2021-03-20 06:37:03 -07:00
Chris Lu
41cc6e661b
fs.mv ensures there are 2 arguments
2021-03-20 06:34:13 -07:00
Chris Lu
b2c1c209a5
refactor
2021-03-20 06:34:13 -07:00
Chris Lu
e9b1853d63
go fmt
2021-03-19 01:31:56 -07:00
Chris Lu
b1a86cf808
s3: copy object to itself
...
fix https://github.com/chrislusf/seaweedfs/issues/1922
2021-03-19 01:31:49 -07:00
Konstantin Lebedev
bf94149920
add line
2021-03-17 23:41:34 +05:00
Konstantin Lebedev
c5705e7a35
dir + slash key does not exist
...
https://github.com/chrislusf/seaweedfs/issues/1917
2021-03-17 23:40:42 +05:00
Konstantin Lebedev
eb54993a4e
Merge branch 'upstreamMaster' into check_chunkviews_mr
...
# Conflicts:
# weed/filer/filechunk_manifest.go
# weed/filer/stream.go
# weed/replication/repl_util/replication_util.go
# weed/util/fasthttp_util.go
2021-03-16 15:29:49 +05:00
Chris Lu
9672f9e1b2
2.34
2021-03-16 03:01:15 -07:00
Chris Lu
69694a17be
reverting 7d57664c2d
2021-03-16 02:59:26 -07:00
Chris Lu
c00dd5e62e
report error if entry not found
2021-03-16 02:55:49 -07:00
Konstantin Lebedev
06da02739d
CheckAllChunkViews() for HEAD requests only
2021-03-16 14:15:17 +05:00
Chris Lu
91a3ac9731
2.33
2021-03-16 00:36:06 -07:00
Chris Lu
4b1ed227d1
revert fasthttp changes
...
related to https://github.com/chrislusf/seaweedfs/issues/1907
2021-03-16 00:33:14 -07:00
Chris Lu
10164d0386
Merge pull request #1905 from wuh-fnst/s3ListCounterErr
...
s3: count correction for the number of files in the dir
2021-03-15 20:54:58 -07:00
Chris Lu
22a07a12d2
[mount] [regression] filer.path
option results in empty mount in 2.32
...
fix https://github.com/chrislusf/seaweedfs/issues/1906
2021-03-15 20:46:25 -07:00
wuh-fnst
050a7f9599
s3: count correction for the number of files in the dir
2021-03-16 09:22:03 +08:00
Konstantin Lebedev
3a3699867b
Status PartialContent for Content-Range response
2021-03-15 23:30:22 +05:00
Konstantin Lebedev
90510e3137
If WriteHeader is not called explicitly, the first call to Write
...
will trigger an implicit WriteHeader(http.StatusOK).
WriteHeader: Only one header may be written. Go does not currently!
2021-03-15 21:22:59 +05:00
Konstantin Lebedev
46b9f5cff4
add debug logging
2021-03-15 19:33:33 +05:00
Konstantin Lebedev
7194a5e7bf
avoid http error: superfluous response.WriteHeader
...
https://github.com/chrislusf/seaweedfs/issues/1838
2021-03-15 18:52:59 +05:00
Chris Lu
9f00f95bfb
2.32
2021-03-14 21:32:00 -07:00
Chris Lu
a4cfffc264
shell: fix moving volume, volume server evacuate
...
fix https://github.com/chrislusf/seaweedfs/issues/1534
2021-03-14 21:29:55 -07:00
Chris Lu
ca100568f7
fix parsing
2021-03-14 20:50:14 -07:00
Chris Lu
352ba23f83
revert previous change
...
revert 29e62aba00
2021-03-14 20:49:56 -07:00
Chris Lu
29e62aba00
possible fix for volume balance
...
address https://github.com/chrislusf/seaweedfs/issues/1534
2021-03-14 20:23:19 -07:00
Chris Lu
35d939a1d2
go fmt
2021-03-14 13:21:02 -07:00
Chris Lu
2d4c2db81d
filer: leveldb, rocksdb auto create store directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1901
2021-03-14 13:20:14 -07:00
Chris Lu
6d3a96eb56
filer: mysql2, postgres2 trigger actions on bucket creation and deletion
...
fix https://github.com/chrislusf/seaweedfs/issues/1877
2021-03-13 22:07:39 -08:00
Chris Lu
cb423312a4
prevent nil volume message
2021-03-13 11:05:29 -08:00
Chris Lu
972327f966
prevent nil volume nm
2021-03-13 11:04:51 -08:00
Chris Lu
2a68ddb963
default to empty host to bind to tcp4 and tcp6
2021-03-12 15:32:51 -08:00
Chris Lu
5fbcaaf37a
fullpath() test is not needed
2021-03-12 09:45:25 -08:00
Chris Lu
7d57664c2d
mount: internals switch to filer.Entry instead of protobuf
2021-03-12 00:36:38 -08:00
wuh-fnst
4c1d945e46
make List correctly judge whether it is the last file
2021-03-12 11:22:26 +08:00
Chris Lu
cca66c7fbe
print out the url
2021-03-11 18:39:45 -08:00
Chris Lu
7f887c78d2
fix
2021-03-11 18:18:16 -08:00
Chris Lu
fcc52d1e4e
shell: add s3.clean.uploads to clean up stale multipart uploads
...
fix https://github.com/chrislusf/seaweedfs/issues/1855
2021-03-11 18:05:24 -08:00
Chris Lu
40dc5ac904
mount: add a bit retry when connecting to filer during startup
2021-03-11 16:36:43 -08:00
Chris Lu
aba47fd9e8
reduce repeated calls to dir.FullPath()
2021-03-11 14:08:20 -08:00
Chris Lu
d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder
...
s3 delete-objects pass OK for fail to delete non-empty folder
2021-03-11 11:01:44 -08:00
Chris Lu
0e35836cc1
Merge pull request #1890 from kmlebedev/ZeroDataSize
...
error if read chunk zero data size
2021-03-11 10:51:48 -08:00
Konstantin Lebedev
58cdcc6d6e
error if urls not found
2021-03-11 23:34:36 +05:00
Chris Lu
79280e1513
escape special characters
...
fix https://github.com/chrislusf/seaweedfs/issues/1884
2021-03-11 09:49:40 -08:00
Konstantin Lebedev
4db21012c1
error if read chunk zero data size
2021-03-11 22:38:59 +05:00
wuheng
828fbf3fb7
s3: "isLast" returns true when the file does not exist
2021-03-11 15:20:50 +08:00
Chris Lu
5d446673a6
fix error printing
2021-03-10 17:13:04 -08:00
Chris Lu
ca546f47e1
s3: escape object key if containing special characters
...
fix https://github.com/chrislusf/seaweedfs/issues/1884
2021-03-10 13:19:28 -08:00
Chris Lu
105fcd7bfc
Merge pull request #1872 from kmlebedev/grpc_auth
...
TLS allowed CommonNames
2021-03-10 11:17:49 -08:00
Konstantin Lebedev
bb3af2d70b
no directoriesWithDeletion
2021-03-10 19:52:41 +05:00
Konstantin Lebedev
15b018da34
s3 delete-objects pass OK for fail to delete non-empty folder
2021-03-10 19:41:35 +05:00
Konstantin Lebedev
348e21a08c
add comments
2021-03-10 14:42:39 +05:00
Konstantin Lebedev
831953c55c
allowed wildcard domain
2021-03-10 14:02:13 +05:00
Konstantin Lebedev
4bf93d6e63
comma-separated
2021-03-10 12:43:13 +05:00
Konstantin Lebedev
0e02f7e258
comma-separated SSL certificate common names
2021-03-10 12:42:44 +05:00
Chris Lu
737bde5ab7
refactoring, remove duplicated code
2021-03-09 23:23:01 -08:00
Chris Lu
5ba4b479f8
properly lock file.entry object
...
fix https://github.com/chrislusf/seaweedfs/issues/1882
2021-03-09 23:08:38 -08:00
Chris Lu
5be83dd4a9
add comments
2021-03-09 14:13:48 -08:00
Chris Lu
9d8ca513d9
refactor
2021-03-09 13:21:26 -08:00
Chris Lu
f5b5e4da2d
2.31
2021-03-09 12:52:16 -08:00
Chris Lu
828f6e9f4d
volume: auto add missing vif files
...
fix https://github.com/chrislusf/seaweedfs/issues/1878
2021-03-09 12:09:32 -08:00
Chris Lu
387c6f4218
Revert "delete the folder object in multi_object_delete"
...
This reverts commit 2e89c8c9ae
.
2021-03-09 10:07:27 -08:00
qieqieplus
5b16820924
fix: collection stats won't update if all volumes expired at same iteration
2021-03-09 18:49:45 +08:00
Konstantin Lebedev
190fada1ef
TLS allowed commonNames
2021-03-08 21:39:44 +05:00
Konstantin Lebedev
c6d3735605
permitCommonNames
...
https://github.com/chrislusf/seaweedfs/issues/1841
https://jbrandhorst.com/post/grpc-auth/
2021-03-08 13:16:17 +05:00
Chris Lu
d888e9a90f
2.30
2021-03-07 15:01:19 -08:00
Chris Lu
60686a0bb7
make tcp optional
2021-03-07 14:45:36 -08:00
Chris Lu
3b2a9c98ef
fix nil
2021-03-07 11:45:35 -08:00
Chris Lu
df72dc206d
stats collect chunk upload retry count
2021-03-07 11:26:15 -08:00
Chris Lu
726edab054
avoid nil when closing an index
...
fix https://github.com/chrislusf/seaweedfs/issues/1870
2021-03-07 11:03:09 -08:00
Chris Lu
bdfed16d42
avoid nil exception
...
fix https://github.com/chrislusf/seaweedfs/issues/1869
2021-03-07 01:49:06 -08:00
Chris Lu
1bd880dcdb
adds tcp writes benchmark
2021-03-06 14:26:27 -08:00
Chris Lu
9d402ebe9f
refactoring
2021-03-06 14:26:27 -08:00
Chris Lu
e9eddfb8a2
remove wrong writes
2021-03-06 14:26:27 -08:00
Chris Lu
8363be8548
trim out trailing return character
2021-03-06 14:26:27 -08:00
Chris Lu
38fc200e56
CRCWriter consistent with CRC
2021-03-06 14:26:27 -08:00
Chris Lu
1444e9d275
migrated multi host connection pool from godropbox package
...
removing unneeded dependencies, which involved etcd versions.
2021-03-06 14:26:27 -08:00
Patrick Schmidt
7413d59750
Fix EC shard count logic
...
This fixes the calculation of the amount of EC shards a node holds.
Previously a global counter was increased, but also used inside the
loop to apply disk usage deltas. This led to wrong absolute numbers.
The fix is to apply only deltas of single EC shards per iteration.
2021-03-05 12:50:58 +01:00
Chris Lu
400de380f4
volume server: support tcp direct put/get/delete
2021-03-05 02:29:38 -08:00
Chris Lu
2e89c8c9ae
delete the folder object in multi_object_delete
...
fix https://github.com/chrislusf/seaweedfs/issues/1857
2021-03-04 14:03:40 -08:00
Chris Lu
c5cb346a08
fix compilation error
2021-03-04 12:36:20 -08:00
Chris Lu
f5cc96f289
skip checking master peers if not starting master
2021-03-04 12:31:26 -08:00
Chris Lu
40ff30b83f
adjust help message
2021-03-03 02:12:46 -08:00
Chris Lu
c0842fe99f
add filer.meta.backup command
2021-03-03 02:02:29 -08:00
Chris Lu
0c5c51eb98
adjust help message
2021-03-02 23:07:39 -08:00
Chris Lu
bd727b7b53
go fmt
2021-03-02 20:59:56 -08:00
Chris Lu
e982b06bcd
support IPv6
2021-03-02 20:59:39 -08:00
Chris Lu
bcf32591b7
reduce possibility of nil entry
2021-03-02 13:33:56 -08:00
Chris Lu
be9c7c21ec
reduce possibility of nil file entry
2021-03-02 11:17:17 -08:00
Chris Lu
02e146f3e1
return err
2021-03-02 08:54:18 -08:00
Chris Lu
5511722420
s3: list permission is needed to list my buckets
...
fix https://github.com/chrislusf/seaweedfs/issues/1837
2021-03-01 12:41:55 -08:00
Chris Lu
c46d16b2f3
skip network error when remote volumes start
2021-03-01 01:20:06 -08:00
Chris Lu
6e43e8ce17
add logs for remote file loading
2021-03-01 00:48:59 -08:00
Chris Lu
015d16f43f
add vif file versions in case loading superblock fails
2021-03-01 00:48:30 -08:00
Chris Lu
d680676d45
skip already loaded backends
2021-03-01 00:47:03 -08:00
Chris Lu
540441fd38
go fmt
2021-02-28 20:34:14 -08:00
Chris Lu
c9722dceb2
adjust log level
2021-02-28 20:26:55 -08:00
Chris Lu
1b1c018165
adjust text
2021-02-28 19:02:43 -08:00
Chris Lu
2cca07b44c
cloud tier to non-AWS s3 gateways
2021-02-28 18:59:09 -08:00
Chris Lu
f29ae0db76
2.29
2021-02-28 18:08:26 -08:00
bingoohuang
5cdff56731
set default env prefix to WEED_
2021-03-01 09:35:23 +08:00
Chris Lu
e52c94640e
filer.backup: added to replace filer.replicate
2021-02-28 16:22:27 -08:00
Chris Lu
014a31d11a
minor
2021-02-28 16:21:09 -08:00
Chris Lu
f2fcb77808
local file sink: create backup files with permission 0755
2021-02-28 16:20:47 -08:00
Chris Lu
984fdd6192
always use non bucket prefixing url
2021-02-28 16:20:13 -08:00
Chris Lu
9abb041763
filer source: support filerProxy mode
2021-02-28 16:19:47 -08:00
Chris Lu
678c54d705
data sink: add incremental mode
2021-02-28 16:19:03 -08:00
Chris Lu
4ff2c5c4c9
rename file
2021-02-28 16:14:21 -08:00
Chris Lu
3b76a51f5f
add help message
2021-02-27 12:15:49 -08:00
Chris Lu
70434df105
use "options"
2021-02-27 12:12:53 -08:00
Chris Lu
d3c31c69a7
avoid confusion: conf and config are too similar
2021-02-27 12:08:09 -08:00
Chris Lu
543c8daa6d
Merge pull request #1844 from kmlebedev/processRangeRequest
2021-02-26 09:18:22 -08:00
Patrick Schmidt
5f7b024891
Show the real disk usage in stats calls
...
Currently the file size of only one volume location is taken into
account in the stats. This commit multiplies the disk usages by the
amount of nodes holding a replica of the volume.
This will yield the expected amount of disk usage and matches the
total size calculations from before.
2021-02-26 13:58:40 +01:00
Konstantin Lebedev
ee21c0042e
log error write entry
2021-02-26 14:18:01 +05:00
Chris Lu
7ba75e3d5a
filer: do not return no content for empty files
...
fix https://github.com/chrislusf/seaweedfs/issues/1831
fix https://github.com/chrislusf/seaweedfs/issues/1830
2021-02-23 12:25:27 -08:00
Chris Lu
9edd964627
volume.tier.move: avoid repeated move for replicated volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-784139348
2021-02-23 03:49:14 -08:00
Chris Lu
37f104f88f
2.28
2021-02-22 22:54:34 -08:00
Chris Lu
90cdf9dcac
avoid conflict with "weed scaffold -config=xxx"
2021-02-22 16:57:27 -08:00
Chris Lu
72b0d9d8c4
avoid unnecessary user home checking
2021-02-22 16:50:56 -08:00
Chris Lu
2270737344
volume: avoid fixed vacuum timeout for large volumes
...
1GB for 3 minutes, about 5.7MB/s
2021-02-22 12:52:37 -08:00
Chris Lu
44bdfb2d15
filer: avoid encryption and compression at the same time
...
fix https://github.com/chrislusf/seaweedfs/issues/1828
2021-02-22 12:22:49 -08:00
Chris Lu
62191b08ea
disk type support custom tags
2021-02-22 02:03:12 -08:00
Chris Lu
30b30b8fe0
volume.tier.move: passing non-empty disk type
2021-02-22 01:59:03 -08:00
Chris Lu
5da63e045e
avoid moving to another server with the same volume id
2021-02-22 01:44:18 -08:00
Chris Lu
6a4546d2c0
shell: add volume.tier.move
2021-02-22 01:30:07 -08:00
Chris Lu
1c233ad986
refactoring
2021-02-22 00:28:42 -08:00
Chris Lu
151c281f36
2.27
2021-02-21 19:29:27 -08:00
Chris Lu
258e93bc86
Merge pull request #1823 from bingoohuang/master
...
feature: support command line arguments in a configuration file
2021-02-20 22:35:18 -08:00
Chris Lu
03c643aa6b
fix test
...
fileSize need to be divided by 8
2021-02-20 21:19:21 -08:00
Chris Lu
7635f6b9fa
disk file avoid file.Stat()
2021-02-20 20:06:06 -08:00
bingoo
1af6c96cd0
fix log help
2021-02-21 10:46:01 +08:00
Chris Lu
a2383b3b12
Revert "Revert "avoid file.Stat()""
...
This reverts commit 7ef4c24f18
.
2021-02-20 12:42:09 -08:00
Chris Lu
7ef4c24f18
Revert "avoid file.Stat()"
...
This reverts commit 98c93ca465
.
2021-02-20 12:39:33 -08:00
Chris Lu
1ad3200094
skip seek() when index file writes
2021-02-20 12:39:25 -08:00
Chris Lu
98c93ca465
avoid file.Stat()
...
avoid one Syscall, but did not help on performance though
2021-02-20 09:44:17 -08:00
bingoohuang
ee082ae402
revert comment temporarily
2021-02-20 16:59:38 +08:00
bingoohuang
ee7cdf3668
revert ErrXyz to ErrorXyz temporarily.
2021-02-20 16:58:08 +08:00
bingoohuang
50df484d86
support command line arguments in a configuration file so we can weed master -config=master.conf
, weed volume -config=volume.conf
and etc.
2021-02-20 16:52:57 +08:00
bingoohuang
f69356f589
Merge pull request #3 from chrislusf/master
2021-02-20 16:45:02 +08:00
Chris Lu
565f7a6e72
Update data_node.go
2021-02-19 14:22:36 -08:00
Chris Lu
a37473ae60
add back volume ids
...
address https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-782339576
2021-02-19 14:22:12 -08:00
Chris Lu
a1210d1e8d
Revert "volume: avoid file.stat(), file.seek() if possible during writes"
...
This reverts commit c78409a598
.
2021-02-19 03:56:27 -08:00
Chris Lu
b961cd6208
add WIP message
2021-02-19 03:39:19 -08:00
Chris Lu
f465d63b5d
grpc should fail when heart beating to master
...
fix https://github.com/chrislusf/seaweedfs/issues/1820
2021-02-19 03:37:29 -08:00
Chris Lu
c576ad04ac
fix volume server display for volumes
2021-02-19 01:38:56 -08:00
bingoohuang
7ffe736d20
renaming to ErrXyz
2021-02-19 15:59:55 +08:00
Chris Lu
c78409a598
volume: avoid file.stat(), file.seek() if possible during writes
2021-02-18 23:22:54 -08:00
Chris Lu
29575dadc5
remove extra files
2021-02-18 19:14:58 -08:00
Chris Lu
73958e357d
add descriptive error if no free volumes
2021-02-18 19:10:20 -08:00
Chris Lu
776f497469
filer: fs.configure should try to read from entry.content also
...
related to https://github.com/chrislusf/seaweedfs/issues/1792
2021-02-18 17:07:02 -08:00
Chris Lu
e1992c83de
webdav adds disktype
2021-02-18 15:45:44 -08:00
Chris Lu
c2ad6f1047
webdav add replication setting
...
fix https://github.com/chrislusf/seaweedfs/issues/1817
2021-02-18 12:15:09 -08:00
Chris Lu
816ef0ed30
add more help message
...
fix https://github.com/chrislusf/seaweedfs/issues/1816
2021-02-18 10:34:49 -08:00
bingoohuang
352ac2f271
Merge remote-tracking branch 'origin/master'
2021-02-18 14:05:51 +08:00
bingoohuang
eab6e31d34
use backticks instead of double quotes to avoid escaped additionally in regex
2021-02-18 14:05:28 +08:00
Chris Lu
3575d41009
go fmt
2021-02-17 20:57:08 -08:00
Chris Lu
6daa932f5c
refactoring to get master function, instead of passing master values directly
...
this will enable retrying later
2021-02-17 20:55:55 -08:00
Chris Lu
dd9f3a0104
add sleep between upload retries
2021-02-17 20:55:11 -08:00
Chris Lu
aa17311063
s3: set filename when uploading file chunks
2021-02-17 20:54:53 -08:00
Chris Lu
3f8b0da677
filer: do not print password on error
...
fix https://github.com/chrislusf/seaweedfs/issues/1809
2021-02-17 02:13:52 -08:00
Chris Lu
cd866664a8
skip JWT if fileId is empty
...
related to https://github.com/chrislusf/seaweedfs/issues/1808
2021-02-16 15:39:12 -08:00
Chris Lu
632c94d438
Merge branch 'extend_to_disk_type'
2021-02-16 11:06:13 -08:00
Chris Lu
0611233f16
adjust printing
2021-02-16 10:55:30 -08:00
Chris Lu
68775d29e3
fix tests
2021-02-16 10:51:03 -08:00
Chris Lu
7403cd43c5
only use "" for hdd to avoid two values for the same thing
2021-02-16 10:50:42 -08:00
Chris Lu
b314d78e97
fix print
2021-02-16 10:48:28 -08:00
Chris Lu
53ca7e66ef
avoid dead lock
2021-02-16 10:48:16 -08:00
Chris Lu
38efc6f572
simplify
2021-02-16 10:47:32 -08:00
Chris Lu
ad70107eb4
ec balance among nodes with HDD
2021-02-16 06:22:50 -08:00
Chris Lu
3097b9a9b7
fix existence checking
2021-02-16 05:59:43 -08:00
Chris Lu
ebf320ec22
avoid divided by zero
2021-02-16 05:59:24 -08:00
Chris Lu
a0c6db361c
avoid nil
2021-02-16 05:33:38 -08:00
Chris Lu
36f95e50a9
avoid possible nil disk info
2021-02-16 05:13:48 -08:00
Chris Lu
43101ccea0
move to the empty nodes first
2021-02-16 04:27:16 -08:00
Chris Lu
38bbef7ec1
avoid nil
2021-02-16 04:16:46 -08:00
Chris Lu
cb9cc29518
volume.list display; fix updating maxVolumeCount for disk
2021-02-16 03:55:24 -08:00
Chris Lu
3fe628f04e
use hdd instead of empty string
2021-02-16 03:03:00 -08:00
Chris Lu
f8446b42ab
this can compile now!!!
2021-02-16 02:47:02 -08:00
Chris Lu
b9b5b932c5
filer: postgres2 mysql2 avoid repeatedly creating tables
2021-02-15 16:19:24 -08:00
Chris Lu
71f0c19515
2.26
2021-02-15 13:38:38 -08:00
Chris Lu
8b169aa47c
s3: Critical - S3 Delete Multiple Objects, delete the bucket
...
fix https://github.com/chrislusf/seaweedfs/issues/1806
2021-02-15 13:38:29 -08:00
Chris Lu
50202d1391
Merge pull request #1805 from LazyDBA247-Anyvision/master
...
better postgresql connection pool management
2021-02-14 22:26:29 -08:00
Chris Lu
8726ef34c1
2.25
2021-02-14 22:23:43 -08:00
LazyDBA247-Anyvision
7f458d5e78
better postgres connection pool management
...
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime )
to enable refresh of stale connections.
2021-02-15 07:45:09 +02:00
Chris Lu
0bc3a1f9e8
disk type only supports hdd and ssd, not ready for random tags yet
2021-02-14 11:38:43 -08:00
LazyDBA247-Anyvision
51b4963e2e
postgres2 & memsql2
...
add escape (quote identifiers) for the dynamic sql
so tables (collections) with special characters will work.
2021-02-14 13:14:36 +02:00
Chris Lu
876dbe3d26
collect disk types
2021-02-13 23:25:16 -08:00
Chris Lu
861f8fab9b
refactoring
2021-02-13 22:34:12 -08:00
Chris Lu
881f05f5a0
adjust help message
2021-02-13 16:13:52 -08:00
Chris Lu
ef76365ec2
adjust help message
2021-02-13 15:47:08 -08:00
Chris Lu
7ce647f27e
support customizable disk type
2021-02-13 15:42:42 -08:00
Chris Lu
4bd8a692d8
disk type can be generic tags
2021-02-13 13:50:14 -08:00
Chris Lu
712b3e9e53
mount: release need to avoid nil reader
...
fix https://github.com/chrislusf/seaweedfs/issues/1803
2021-02-13 13:39:37 -08:00
Chris Lu
4ce56bac08
fix format error
2021-02-12 06:44:26 -08:00
Chris Lu
0604afcda7
mount: remove unnecessary flush on release
2021-02-12 06:01:04 -08:00
Chris Lu
7d9dc3c6a2
use fasthttp lib to read
2021-02-12 05:59:52 -08:00
Chris Lu
487e435679
adjust http max idle connections per host
...
related to https://github.com/chrislusf/seaweedfs/issues/1802
2021-02-12 03:47:15 -08:00
Chris Lu
0f426ce34d
Merge pull request #1801 from kmlebedev/recoveringRabbitMQ
...
Do reconnect to RabbitMQ
2021-02-11 01:05:03 -08:00
Konstantin Lebedev
94eac4f00e
Do reconnect to RabbitMQ
...
https://github.com/google/go-cloud/issues/2958
https://github.com/chrislusf/seaweedfs/issues/1773
2021-02-11 13:59:36 +05:00
Chris Lu
885ca34748
volume: fail fast if idx files are missing
...
fix https://github.com/chrislusf/seaweedfs/issues/1796
2021-02-11 00:44:40 -08:00
Chris Lu
a0e84c4fbc
go fmt
2021-02-10 23:41:05 -08:00
Chris Lu
c959aaa19b
mount: load directory based on correct filer path
...
fix https://github.com/chrislusf/seaweedfs/issues/1799
2021-02-10 23:14:18 -08:00
Konstantin Lebedev
0cfed8c3cb
restart replication if RabbitMQ connection closed
...
https://github.com/google/go-cloud/issues/2958
2021-02-10 15:29:05 +05:00
Chris Lu
770393a48c
volume: add capability to change disk type when moving a volume
2021-02-09 23:58:08 -08:00
Chris Lu
821c46edf1
Merge branch 'master' into support_ssd_volume
2021-02-09 11:37:07 -08:00
Chris Lu
a833021132
fix refactoring left over
2021-02-07 23:03:03 -08:00
Chris Lu
5e4b5109dd
2.24
2021-02-07 22:29:43 -08:00
Chris Lu
15c60cbb26
close the grpc connection after 10 hours
...
related to https://github.com/chrislusf/seaweedfs/issues/1782
2021-02-07 03:50:01 -08:00
bingoohuang
7256902fb0
fix typo offset.ToAcutalOffset to offset.ToActualOffset
2021-02-07 12:11:51 +08:00
bingoohuang
94ea3bd3a5
renaming NeedleMapType to NeedleMapKind
2021-02-07 09:00:03 +08:00
Chris Lu
c3af72d950
fier store: fix elastic search regression
...
fix https://github.com/chrislusf/seaweedfs/issues/1774
2021-02-04 01:30:14 -08:00
Konstantin Lebedev
2fb4c60540
fix log warn
2021-02-03 21:34:32 +05:00
Konstantin Lebedev
24ec17219b
RabbitMQ delay retry with Dead Letter Exchange
...
https://github.com/chrislusf/seaweedfs/issues/1773
https://github.com/google/go-cloud/issues/2952
2021-02-03 21:32:30 +05:00
Chris Lu
a331bbb3ae
filer: should return 204 on DELETE to nonexistent file
...
related to
https://github.com/chrislusf/seaweedfs/issues/1776
https://github.com/chrislusf/seaweedfs/issues/1160
2021-02-03 00:40:31 -08:00
Chris Lu
5353e38469
S3: should return 204 on DELETE to nonexistent file
...
fix https://github.com/chrislusf/seaweedfs/issues/1776
2021-02-03 00:35:44 -08:00
Chris Lu
2396ac234c
filer: fix elastic search pagination
...
possible fix for https://github.com/chrislusf/seaweedfs/issues/1774
2021-02-02 11:34:12 -08:00
Chris Lu
609daaf387
s3: DeleteMultipleObjectsHandler clean up leftover empty folders
...
fix https://github.com/chrislusf/seaweedfs/issues/1772
2021-02-01 10:49:17 -08:00
Chris Lu
d67ccb66c0
webdav: can start together with "weed server" or "weed filer"
2021-01-31 22:16:52 -08:00
Chris Lu
76430790b9
webdav: cache to version specific folder
2021-01-31 22:02:03 -08:00
Chris Lu
318a3d2efc
2.23
2021-01-31 20:18:06 -08:00
Chris Lu
4be51c0701
filer: leveldb and hbase may miss files when listing large directories more than 1024
...
fix https://github.com/chrislusf/seaweedfs/issues/1768
2021-01-31 20:11:44 -08:00
Chris Lu
1102ae32c4
fix concurrent map reads
2021-01-31 18:26:26 -08:00
Chris Lu
314dc1c957
filer: etcd store fix listing
...
fix https://github.com/chrislusf/seaweedfs/issues/1767
2021-01-30 13:38:44 -08:00
Chris Lu
d475c89fcc
go fmt
2021-01-28 15:23:46 -08:00
Chris Lu
19295600f9
mount: change option name to volumeServerAccess, with publicUrl and filerProxy modes
2021-01-28 15:23:16 -08:00
Chris Lu
990fa69bfe
add back AdjustedUrl() related code
2021-01-28 14:36:29 -08:00
Chris Lu
9292796ec2
fmt
2021-01-28 14:29:14 -08:00
Chris Lu
c2bf1a88ac
delete from the deepest directory first when checking empty folders
2021-01-28 14:28:40 -08:00
Chris Lu
e9d8201925
s3: batch purge empty folders
2021-01-28 13:20:06 -08:00
Chris Lu
cf252fc0cd
mount: report error when Flush()
...
The error in Release() is not working.
See https://github.com/jaderhs/libfuse/blob/master/FAQ
related to https://github.com/chrislusf/seaweedfs/issues/1765
2021-01-28 04:46:37 -08:00
Chris Lu
9a06c35da4
replicate: incremental sink only contains new and updated files
...
address da08402ba2
2021-01-28 02:39:22 -08:00
Chris Lu
da08402ba2
replicate: use creation time for local incremental file sink
...
related to https://github.com/chrislusf/seaweedfs/pull/1762
2021-01-28 02:17:41 -08:00
Chris Lu
822f1ade9d
Merge pull request #1762 from kmlebedev/backupsink
...
replication to create time date directory
2021-01-28 02:03:33 -08:00
Konstantin Lebedev
be1062b7fc
rename in scaffold
2021-01-28 14:59:20 +05:00
Konstantin Lebedev
02fdc0a333
rename backup to local_incremental and use mtime
2021-01-28 14:56:13 +05:00
Chris Lu
5f72c388bc
Merge pull request #1764 from danlsgiga/handle-umask
...
Use provided umask on mount
2021-01-28 01:09:20 -08:00
Daniel Santos
d38613c49a
match upstream
2021-01-27 20:56:41 -07:00
Chris Lu
b81956bcb5
mount: invalidate kernel cache when mounted to a filer path
...
fix https://github.com/chrislusf/seaweedfs/issues/1752#issuecomment-768178422
2021-01-27 10:28:37 -08:00
Konstantin Lebedev
6b54ff9912
replication to create time date directory
2021-01-27 15:01:33 +05:00
Chris Lu
f20ec82a28
update help message
...
see https://github.com/chrislusf/seaweedfs/issues/1758#issuecomment-768123224
2021-01-27 00:48:31 -08:00
Chris Lu
1045ab8b07
avoid error message
2021-01-27 00:21:59 -08:00
Chris Lu
0488c84bea
adjust help message
2021-01-27 00:17:52 -08:00
Konstantin Lebedev
3634811408
replication to todays date directory
2021-01-27 12:45:58 +05:00
Chris Lu
5e07afb0f0
Merge pull request #1759 from kmlebedev/sink.local
...
replication to local disk storage
2021-01-26 22:31:54 -08:00
Chris Lu
b3f66199db
shell: volume.fix.replication adds collectionPattern with wildcard characters
...
fix https://github.com/chrislusf/seaweedfs/issues/1758
2021-01-26 22:30:39 -08:00
Daniel Santos
0c64d60fdb
Fix log message with correct mode
2021-01-26 21:45:52 -07:00
Daniel Santos
0aaaa9b470
Fix var assignment
2021-01-26 21:06:08 -07:00
Daniel Santos
5a761bb7a4
Enforce umask correctly
2021-01-26 20:42:10 -07:00
Chris Lu
ad2a20c8a5
notification add ack and nack
2021-01-26 11:08:44 -08:00
Konstantin Lebedev
612b7975a1
replication to local disk storage
2021-01-26 22:50:25 +05:00
Chris Lu
3a1d3d3413
mount: properly invalidate kernel node cache entry
...
fix https://github.com/chrislusf/seaweedfs/issues/1752
2021-01-26 02:50:53 -08:00
Chris Lu
cc1f3907ff
filer.sync: default filerProxy to true
2021-01-25 10:14:42 -08:00
Chris Lu
62ebb917e0
mount: avoid the need to specify different cache directory for different mount
...
fix https://github.com/seaweedfs/seaweedfs-csi-driver/issues/11
2021-01-25 09:10:20 -08:00
Chris Lu
17f7c1c43f
Merge branch 'mount_from_outside_cluster'
2021-01-24 19:03:02 -08:00
Chris Lu
00707ec00f
mount: outsideContainerClusterMode proxy through filer
...
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
2021-01-24 19:01:58 -08:00
Chris Lu
9bcb28a3ea
avoid possible nil reader
...
fix https://github.com/chrislusf/seaweedfs/issues/1754
2021-01-24 17:19:35 -08:00
Chris Lu
2c5eac5705
2.22
2021-01-24 14:06:09 -08:00
Chris Lu
6ca10725b8
Revert "mount: when outside cluster network, use filer as proxy to access volume servers"
...
This reverts commit 096e088d7b
.
2021-01-24 03:15:19 -08:00
Chris Lu
096e088d7b
mount: when outside cluster network, use filer as proxy to access volume servers
2021-01-24 01:41:38 -08:00
Chris Lu
ccbdb38c89
s3: fix for listing objects if more than 1000 in the folder
...
many existing tools have max key set to 1000
2021-01-24 00:27:52 -08:00
Chris Lu
80b8692688
filer.sync: replicate outside of either cluster, only need to see filers
2021-01-24 00:01:44 -08:00
Chris Lu
f8dbb03bdd
filer: append operation returns final file size
2021-01-23 04:19:49 -08:00
Chris Lu
759482e2ac
filer: proxy all http methods to volume servers
2021-01-23 03:52:38 -08:00
Chris Lu
937cfacc01
filer: add "proxyToFileId" to reverse proxy to a volume server
2021-01-23 03:43:48 -08:00
Chris Lu
20ef3bb8d4
mount: use direct_io to avoid OS page cache
...
fix https://github.com/chrislusf/seaweedfs/issues/1752
2021-01-22 22:39:46 -08:00
Chris Lu
711c3f3939
filer: fix http status 206 setting
...
fix https://github.com/chrislusf/seaweedfs/issues/1753
2021-01-22 00:26:30 -08:00
Chris Lu
024c186a48
filer.meta.tail: output json format for easier parsing
2021-01-21 22:39:25 -08:00
Chris Lu
84f05787f8
mount: limit background requests and congestion threshold
2021-01-20 23:30:14 -08:00
Chris Lu
f98817cfe6
filer: support appending to a file
2021-01-20 13:40:32 -08:00
Chris Lu
1b8e3da295
refactoring
2021-01-20 12:59:54 -08:00
Chris Lu
1a3f859c58
filer store: postgres configuration adjust text
2021-01-19 18:12:07 -08:00
Chris Lu
b434f7e4e0
filer store: postgres adjust default config, add optioanl schema
2021-01-19 18:10:36 -08:00
Chris Lu
d5add83e85
filer store: add postgres2
2021-01-19 18:07:29 -08:00
Chris Lu
52a8f1470e
filer store: add mysql2
2021-01-19 17:21:50 -08:00
Chris Lu
93b3adba98
fix bucket creation
2021-01-19 15:55:51 -08:00
Chris Lu
fa0c8d5283
fix error
2021-01-19 14:05:48 -08:00
Chris Lu
4c5b752b04
restructuring sql stores
2021-01-19 13:53:16 -08:00
Chris Lu
96354208c5
scaffold: add YugabyteDB
2021-01-19 12:35:14 -08:00
Chris Lu
ca8f793978
refactor: prepare for bucket specific tables
2021-01-19 12:34:58 -08:00
Chris Lu
90ce1eec19
add more input params
2021-01-19 11:32:55 -08:00
Chris Lu
f13c082583
add memsql to comments
2021-01-19 11:31:55 -08:00
Chris Lu
e439b65e38
fix test
2021-01-18 01:48:00 -08:00
Chris Lu
c7197470fd
2.21
2021-01-18 01:25:04 -08:00
Chris Lu
5a3386e39b
Revert "Revert "mount: fake support for socket/block/character/fifo devices""
...
This reverts commit 3d5cb7eb86
.
2021-01-18 01:15:07 -08:00
Chris Lu
67faa56920
Revert "Revert "mount: add more entry type""
...
This reverts commit 4085b79d38
.
2021-01-18 01:14:58 -08:00
Chris Lu
7119e7b491
mount: ensure reading from dirty pages
2021-01-18 01:14:42 -08:00
Chris Lu
de876c795d
minor fix
2021-01-18 01:14:27 -08:00
Chris Lu
4085b79d38
Revert "mount: add more entry type"
...
This reverts commit d7d907be27
.
2021-01-18 00:19:19 -08:00
Chris Lu
3d5cb7eb86
Revert "mount: fake support for socket/block/character/fifo devices"
...
This reverts commit 61ef2d8658
.
2021-01-18 00:18:57 -08:00
Chris Lu
61ef2d8658
mount: fake support for socket/block/character/fifo devices
...
to pass pjdfstest. A distributed file system can not really support these things anyway.
2021-01-17 23:51:10 -08:00
Chris Lu
d7d907be27
mount: add more entry type
2021-01-17 23:46:39 -08:00
Chris Lu
389426bbb7
s3: listing may repeat on the edge
2021-01-17 22:59:31 -08:00
Chris Lu
09f49d1c04
refactoring
2021-01-16 19:52:15 -08:00
Chris Lu
a4063a5437
add stream list directory entries
2021-01-15 23:56:24 -08:00
Chris Lu
01dc8a43ba
fix rocksdb
2021-01-15 18:31:29 -08:00
Chris Lu
16ad74f477
go fmt
2021-01-14 23:11:27 -08:00
Chris Lu
f002e668de
change limit to int64 in case of overflow
2021-01-14 23:10:37 -08:00
Chris Lu
19e52fd414
pass along hasMore
2021-01-14 22:49:29 -08:00
Chris Lu
c0bcf6f6e1
implement sql changes
2021-01-14 22:44:22 -08:00
Chris Lu
5d4568b91f
implement elastic changes
2021-01-14 22:42:25 -08:00
Chris Lu
893cbc8482
implement c* changes
2021-01-14 22:38:34 -08:00
Chris Lu
5ef43b9b09
implement etcd changes
2021-01-14 22:35:56 -08:00
Chris Lu
54527f0326
implement hbase changes
2021-01-14 22:34:35 -08:00
Chris Lu
b5ceffe188
implement leveldb changes
2021-01-14 22:33:05 -08:00
Chris Lu
c64bfb0e2e
implement mongodb changes
2021-01-14 22:28:51 -08:00
Chris Lu
34a846009d
implement rocksdb changes
2021-01-14 22:24:26 -08:00
Chris Lu
d2ece29fff
implement redis changes
2021-01-14 22:21:31 -08:00
Chris Lu
9a50dbcda0
chagned api
2021-01-14 22:18:38 -08:00
Chris Lu
e1c7bc66f9
change interface
2021-01-14 22:17:49 -08:00
Chris Lu
4aaa97c789
skip not found error in case of race condition
2021-01-14 19:56:14 -08:00
Chris Lu
d0826c2f62
mount: avoid showing "topics" folder
...
fix https://github.com/chrislusf/seaweedfs/issues/1325
2021-01-14 19:03:15 -08:00
Chris Lu
698f58f7c4
filer, s3: add http status 206 as late as possible
...
fix https://github.com/chrislusf/seaweedfs/issues/1746
2021-01-14 02:59:40 -08:00
Chris Lu
fd1d8a2a09
fix compilation
2021-01-14 00:06:15 -08:00
Chris Lu
ae812c07d6
Merge pull request #1743 from LazyDBA247-Anyvision/master
...
MYSQL - Better Connection Pool Control
2021-01-14 00:04:44 -08:00
Chris Lu
f20743bb38
remove WIP status
2021-01-13 22:21:34 -08:00
LazyDBA247-Anyvision
8eed763b97
better mysql connection pool management
...
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime )
to enable refresh of connections.
2021-01-14 08:14:21 +02:00
Chris Lu
f17aa1d06c
skip directory loop if dropping the bucket
2021-01-13 13:49:04 -08:00
Chris Lu
ca73013453
leveldb3: ensure repeatable adding removing buckets
2021-01-13 13:48:48 -08:00
Chris Lu
e2c7e3fe6d
better locks
2021-01-13 13:20:33 -08:00
Chris Lu
819a85f59a
avoid viper concurrent access
2021-01-13 13:14:52 -08:00
Chris Lu
3fb2ed9093
filer.meta.tail: optionally submit metadata changes to ElasticSearch
2021-01-13 03:31:37 -08:00
Chris Lu
c5df2577f5
rename
2021-01-13 00:31:19 -08:00
Chris Lu
90c5077610
rename "weed watch" to "weed filer.meta.tail"
2021-01-12 18:48:01 -08:00
Chris Lu
5cd9c9e642
remove println
2021-01-12 18:47:38 -08:00
Chris Lu
def6d8e565
fix test
2021-01-12 12:32:54 -08:00
Chris Lu
1c7faf1100
Revert "close opened file"
...
This reverts commit 2880160113
.
2021-01-12 11:07:14 -08:00
Chris Lu
2880160113
close opened file
2021-01-12 11:03:08 -08:00
Chris Lu
0ef43a23a7
go fmt
2021-01-12 02:30:12 -08:00
Chris Lu
629c9962e7
filer: add leveldb3
...
support dedicated leveldb instance for each bucket
2021-01-12 02:29:44 -08:00
Chris Lu
361043e6c1
filer store: leveldb2 fix nil entry error if not found
2021-01-12 02:28:57 -08:00
Chris Lu
cfb9342a15
avoid concurrent map updates to viper
2021-01-12 02:28:13 -08:00
Chris Lu
38d516251e
Revert "mount: avoid fatal error"
...
This reverts commit a4f0bac9b7
.
2021-01-11 13:50:15 -08:00
Chris Lu
a4f0bac9b7
mount: avoid fatal error
2021-01-11 13:26:59 -08:00
Chris Lu
8a7302ce37
s3: KeyCount is zero
...
fix https://github.com/chrislusf/seaweedfs/issues/1741
2021-01-11 03:50:45 -08:00
Chris Lu
2c7148cd55
redis cluster: do not enable route by latency by default
2021-01-11 02:43:53 -08:00
Chris Lu
acc0d5cd2c
Revert "Revert "filer store: redis module upgrade""
...
This reverts commit dd0171e35c
.
2021-01-11 02:30:19 -08:00
Chris Lu
dd0171e35c
Revert "filer store: redis module upgrade"
...
This reverts commit 40161c9cec
.
2021-01-11 02:09:18 -08:00
Chris Lu
394513f598
filer: ensure seamless meta data updates
2021-01-11 02:08:55 -08:00
Chris Lu
6e12a3a490
refactoring
2021-01-11 02:08:26 -08:00
Chris Lu
62ee484d12
refactoring
2021-01-11 00:03:13 -08:00
Chris Lu
1efb51ba84
filer: change to saveToFilerLimit from cacheToFilerLimit
...
short circuit saving small files to volume server
2021-01-10 23:14:46 -08:00
Chris Lu
f0d3b3bf93
2.20
2021-01-09 22:55:22 -08:00
Chris Lu
d3d3f2fb9b
mount: default to 128 concurrent writers
2021-01-09 22:53:37 -08:00
Chris Lu
90b117acf1
update ccache version
2021-01-08 02:17:43 -08:00
Chris Lu
e2e0e30a51
filer: redis default to false for useReadOnly and routeByLatency
2021-01-08 01:12:44 -08:00
Chris Lu
40161c9cec
filer store: redis module upgrade
2021-01-08 01:08:44 -08:00
Chris Lu
30efaa66c6
server: add option to disable master
2021-01-07 02:35:47 -08:00
Chris Lu
07f4703bfc
Merge pull request #1738 from qieqieplus/rocksdb
...
fix rocksdb crash when list directory
2021-01-07 02:26:41 -08:00
qieqieplus
0f18592315
alter style
2021-01-07 18:21:48 +08:00
qieqieplus
c7e7431fd3
fix seek lastkey may reach EOF
2021-01-07 17:07:56 +08:00
Chris Lu
e327385644
go fmt
2021-01-06 04:22:00 -08:00
Chris Lu
2b76854641
add "weed filer.cat" to read files directly from volume servers
2021-01-06 04:22:00 -08:00
Chris Lu
95ecf0c72f
Merge pull request #1735 from qieqieplus/rocksdb
...
ignore decode error for non-entry data
2021-01-06 02:22:18 -08:00
qieqieplus
0764fccde7
ignore decode error for non-entry data
2021-01-06 17:42:43 +08:00
Chris Lu
ab57767d4b
Merge pull request #1730 from qieqieplus/rocksdb
...
impl: TTL per entry for rocksdb; fix package name
2021-01-05 13:24:45 -08:00
qieqieplus
b2e50f602f
fix time comparison
2021-01-05 19:36:40 +08:00
qieqieplus
bbae13e415
impl: TTL per entry for rocksdb; fix package name
2021-01-05 17:58:40 +08:00
henry
b42c5b71c1
Seaweed custom header are not visible to Vue or javascript
2021-01-05 13:01:29 +08:00
henry
1309dfe612
Seaweed custom header are not visible to Vue or javascript
2021-01-05 12:58:46 +08:00
henry
942a30f04c
remove sort
2021-01-05 12:24:32 +08:00
henry
a3c432dadc
Merge branch 'master' of https://github.com/fuyouyshengwu/seaweedfs
2021-01-05 11:17:29 +08:00
qieqieplus
ebb223c190
fix for rocksdb kv
2021-01-04 21:21:46 +08:00
chenqieqie
40b3207f18
fix #1726
2021-01-04 18:48:55 +08:00
henry
97a94eddab
Merge branch 'master' of https://github.com/fuyouyshengwu/seaweedfs
2021-01-04 14:50:01 +08:00
Chris Lu
2ce86f308e
2.19
2021-01-03 17:24:02 -08:00
Chris Lu
9c9ba3c209
nil related
...
related to https://github.com/chrislusf/seaweedfs/issues/1676
2021-01-03 12:25:58 -08:00
Chris Lu
4b5adc60b4
upload result changes to the right output variable
2021-01-03 01:44:22 -08:00
Chris Lu
039ad101b1
manifest file also need to detect compression
...
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753585046
2021-01-03 01:42:21 -08:00
Chris Lu
5f538c5f9d
Merge branch 'add_rocksdb'
2021-01-03 00:48:56 -08:00
Chris Lu
bdd166c031
adjust readme
2021-01-03 00:47:58 -08:00
Chris Lu
a04d8927a8
add scaffold
2021-01-03 00:44:52 -08:00
Chris Lu
4250a3a675
upload: add read default collection from master
...
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753576071
2021-01-02 23:43:01 -08:00
Chris Lu
f4d87f8b9d
works
...
performance seems about the same as leveldb for insert entries
2021-01-02 23:32:58 -08:00
Chris Lu
2c3c2c27d7
separate prefix from namePattern
...
fix https://github.com/chrislusf/seaweedfs/issues/1722
2021-01-01 20:23:23 -08:00
Peter Cai
cd2a9865ed
filer: respect Content-Type on PUT
...
This is needed for the S3 gateway to conform to the S3 protocol -- the
server should always respect the Content-Type set by client.
2020-12-31 11:32:45 +08:00
Chris Lu
3433accb1b
weed upload: add TTL
2020-12-30 15:46:07 -08:00
henry
14ddd15508
1 Distinguish between folders and common files, folders are listed first
...
2 Add some custom headers that need to be exposed
2020-12-31 00:03:22 +08:00
Chris Lu
adef4ddc87
mount: load security keys
...
fix https://github.com/chrislusf/seaweedfs/issues/1714
2020-12-29 11:51:38 -08:00
Chris Lu
89977123b6
shell: add shell.toml support
...
fix https://github.com/chrislusf/seaweedfs/issues/1664
2020-12-28 15:07:18 -08:00
Chris Lu
d2ead72918
2.18
2020-12-28 12:37:10 -08:00
Chris Lu
2e6cf541e7
filer: avoid deletion error when item is not found
2020-12-28 00:18:37 -08:00
Chris Lu
6d2c979999
s3: fix regression
...
fix https://github.com/chrislusf/seaweedfs/issues/1707
2020-12-27 21:09:45 -08:00
Chris Lu
90df52d001
s3: add command option "allowEmptyFolder" to filer and server
2020-12-27 21:09:06 -08:00
Chris Lu
da7e5aaa65
filer: optimize for less number of directory lookup
...
bottom up directory lookup
2020-12-27 20:46:17 -08:00
henry
7458ff9523
Solve the Chinese name disorder of download file
2020-12-28 11:28:33 +08:00
Chris Lu
561a2ff0bc
s3: add option for "alllowEmptyFolder"
2020-12-27 10:50:27 -08:00
Chris Lu
aa020ee7e7
s3: restore V2 auth type
...
The added headers were included in calculating the signatures, failing the comparison
2020-12-26 22:01:16 -08:00
Chris Lu
da87f6b265
remove unused code
2020-12-26 15:21:12 -08:00
Chris Lu
0a7c5f85a9
filer: add namePattern to search in current folder
2020-12-26 15:05:31 -08:00
Chris Lu
515b7632d7
2.17
2020-12-25 23:58:22 -08:00
Chris Lu
0a067944cc
filer: add retries during volume moving
...
fix https://github.com/chrislusf/seaweedfs/issues/1704
2020-12-25 02:32:55 -08:00
Chris Lu
090f85be4b
s3: support config action Admin:bucket
2020-12-25 00:38:56 -08:00
Chris Lu
75613b2cc7
Merge branch 'add_hbase'
2020-12-24 12:13:05 -08:00
Chris Lu
0e016bc7bd
hbase add ttl
2020-12-24 12:10:35 -08:00
Chris Lu
8e48a235e2
s3: avoid looping if the directory is empty
...
fix https://github.com/chrislusf/seaweedfs/issues/1701
2020-12-24 11:34:52 -08:00
Chris Lu
1620de08ae
added more logs
2020-12-24 01:52:06 -08:00
Chris Lu
a09cd28986
add debug messages
2020-12-24 01:42:15 -08:00
Chris Lu
75c6edba9e
filer: hbase add enabled flag
2020-12-24 00:19:16 -08:00
Chris Lu
9bf6c10505
fix prefix
2020-12-24 00:18:59 -08:00
Chris Lu
2fea8cfa0f
fix compilation
2020-12-23 23:54:15 -08:00
Chris Lu
53bc1ea25b
fix compilation
2020-12-23 23:53:46 -08:00
Chris Lu
b5e2be635a
adjust for directory listing
2020-12-23 23:49:22 -08:00
Chris Lu
c4a202ec41
fix wrong column family
2020-12-23 23:23:05 -08:00
Chris Lu
c3d1b3b5aa
hook up
2020-12-23 21:49:01 -08:00
Chris Lu
94e3757c08
fix
2020-12-23 21:45:16 -08:00
Chris Lu
64c48c9724
seems compiling
2020-12-23 21:44:53 -08:00
Chris Lu
15da5834e1
Merge branch 'master' into support_ssd_volume
2020-12-23 14:37:37 -08:00
Chris Lu
3be3635799
just log error when checking empty dir failed
2020-12-23 14:36:48 -08:00
Chris Lu
0ca9d89589
s3: break loop if error
...
fix #1701
2020-12-23 14:34:59 -08:00
Chris Lu
132f275d04
s3: log errors instead of stopping when checking isDirectoryAllEmpty has error
2020-12-23 12:21:21 -08:00
Chris Lu
db62090f88
s3: break loop if error
...
fix https://github.com/chrislusf/seaweedfs/issues/1701
2020-12-23 12:14:51 -08:00
Baptiste Mille-Mathias
385a4dec30
[shell] Clarify bucket replication setting order
2020-12-23 09:35:24 +01:00
Chris Lu
1c7e1295dc
Merge branch 'master' into support_ssd_volume
2020-12-22 17:44:52 -08:00
Chris Lu
da134a2eb7
minor
2020-12-22 17:43:13 -08:00
Chris Lu
beb3b8ddb0
shell: change bucket.list to s3.bucket.list, same for create and delete
2020-12-22 17:40:55 -08:00
Chris Lu
4cf3176cbb
filer first, master second
2020-12-22 17:34:44 -08:00
Chris Lu
a1009e8044
filer: add -defaultStoreDir so that filer.toml can be skipped
...
fix https://github.com/chrislusf/seaweedfs/issues/1659
2020-12-22 17:33:40 -08:00
Chris Lu
a0990b929d
minor
2020-12-22 17:33:40 -08:00
Baptiste Mille-Mathias
0ed5345954
[shell] Add more information to bucket.create
...
State the replication setting that will be honor if not defined.
2020-12-22 17:18:59 +01:00
Chris Lu
97e3432dfe
avoid wrong error
...
fix https://github.com/chrislusf/seaweedfs/issues/1691
2020-12-22 02:46:24 -08:00
Chris Lu
4f31c1bb94
go fmt
2020-12-22 02:34:08 -08:00
Chris Lu
6c4f32d173
remove unused code
2020-12-22 02:33:31 -08:00
Chris Lu
90d785a15f
filer: redis, redis cluster, cassandra support super large directory
2020-12-22 02:26:05 -08:00
Chris Lu
fe46411cd4
handle directory listing edge cases
2020-12-21 23:37:43 -08:00
Chris Lu
c74bede730
handle listing directories
2020-12-21 23:19:05 -08:00
Chris Lu
adf8cb4000
filer: path-specific stores trim out common prefixes
2020-12-21 22:57:13 -08:00
Chris Lu
0823bde8d8
refactor: split into two files
2020-12-21 22:05:15 -08:00
Chris Lu
8ab70ccf76
refactor
2020-12-21 21:46:32 -08:00
Chris Lu
65cc6dc636
refactor to dedicated function
2020-12-21 21:18:34 -08:00
Chris Lu
488c2680e8
change default filer store root
2020-12-21 21:15:22 -08:00
Chris Lu
f5a19a87be
filer: report error if the path specific store is not found
...
related https://github.com/chrislusf/seaweedfs/issues/1691
2020-12-21 12:43:52 -08:00
Chris Lu
6912bf94ae
2.16
2020-12-20 15:30:12 -08:00
Chris Lu
41c0f3ad24
filer: support path-specific filer store
2020-12-19 01:27:09 -08:00
Chris Lu
2260864b45
a cleaner fix
2020-12-18 10:39:30 -08:00
Chris Lu
b7fef8f693
fix when a volume server has only hdd volumes
2020-12-18 10:34:16 -08:00
Chris Lu
0d5683fb0e
todo: load path-specific store from filer.toml
2020-12-18 03:05:42 -08:00
Chris Lu
3269fd7eaf
filer: use store by path
2020-12-18 02:57:49 -08:00
Chris Lu
e605f1e001
refactor, add getActualStore() by path
2020-12-18 02:55:00 -08:00
Chris Lu
23903aa95b
refactoring
2020-12-18 02:35:45 -08:00
Chris Lu
4e6b316913
assert FilerStoreWrapper is VirtualFilerStore
2020-12-18 02:17:06 -08:00
Chris Lu
9abf016af1
Merge branch 'master' into support_ssd_volume
2020-12-17 22:30:19 -08:00
Chris Lu
738c4fd203
fix logging when error
...
address https://github.com/chrislusf/seaweedfs/pull/1685#issuecomment-747845601
2020-12-17 19:37:46 -08:00
Chris Lu
d9e8479c06
adjust UI max count
2020-12-17 13:47:51 -08:00
Chris Lu
3cdf5945a2
adjust UI
2020-12-17 13:37:00 -08:00
Chris Lu
f696a2b2a7
assign volumes based on disk type
2020-12-17 13:25:05 -08:00
Chris Lu
daa8157fc2
Merge branch 'master' into support_ssd_volume
2020-12-17 13:05:20 -08:00
Chris Lu
e2076201d7
volume: avoid reprocessing the same volume
...
fix https://github.com/chrislusf/seaweedfs/issues/1682
2020-12-17 13:03:39 -08:00
Chris Lu
986cbdf7d9
Revert "Merge pull request #1683 from qieqieplus/master"
...
This reverts commit 8cb67952db
, reversing
changes made to 200e56215a
.
2020-12-17 12:46:42 -08:00
Chris Lu
f56e6d231e
Revert "Merge pull request #1683 from qieqieplus/master"
...
This reverts commit 8cb67952db
, reversing
changes made to 200e56215a
.
2020-12-17 12:46:20 -08:00
Chris Lu
1bf22c0b5b
go fmt
2020-12-16 09:14:05 -08:00
Chris Lu
1d88865869
passing disk type along
2020-12-16 09:10:14 -08:00
Chris Lu
23014b6810
Merge branch 'master' into support_ssd_volume
2020-12-16 08:26:51 -08:00
Chris Lu
b789767cd1
filer: return http status 499 for client closed connection
...
fix https://github.com/chrislusf/seaweedfs/issues/1684
2020-12-16 08:18:00 -08:00
chenqieqie
45f902a9b7
smaller critical section
2020-12-16 20:23:51 +08:00
chenqieqie
4e58a4f24e
fix race condition when loading volumes concurrently
2020-12-16 18:49:10 +08:00
Chris Lu
3c7f9633eb
show error a little bit more obviously
...
fix https://github.com/chrislusf/seaweedfs/issues/1678
2020-12-14 09:39:33 -08:00
Chris Lu
97dcae827b
Merge branch 'master' into support_ssd_volume
2020-12-14 01:27:31 -08:00
Chris Lu
3d47c38262
collection.delete requires _default_ as the default empty collection name
...
fix https://github.com/chrislusf/seaweedfs/issues/1677
2020-12-14 01:05:20 -08:00
Chris Lu
2e8dba571b
adjust volume server UI
2020-12-14 00:51:57 -08:00
Chris Lu
bf5873022d
Merge branch 'master' into support_ssd_volume
2020-12-14 00:32:17 -08:00
Chris Lu
b7e3ca9172
Merge pull request #1650 from kmlebedev/masterServerEnableUI
...
security master use access.ui
2020-12-14 00:31:51 -08:00
Chris Lu
021358749b
better detect IP address
...
following advice from https://github.com/chrislusf/seaweedfs/issues/1671
2020-12-14 00:30:20 -08:00
Konstantin Lebedev
c4459249ef
ui interface is always on
2020-12-14 13:24:41 +05:00
Chris Lu
5cffddebb9
add logging
2020-12-14 00:11:52 -08:00
Chris Lu
933f2f4cfd
find location with matching disk type
2020-12-13 23:39:00 -08:00
Chris Lu
94525aa0fd
allocate volume by disk type
2020-12-13 23:08:21 -08:00
Chris Lu
f6a419c26c
disk type configurable for each folder
2020-12-13 22:49:56 -08:00
Chris Lu
7c52a35942
refactor
2020-12-13 22:29:52 -08:00
Chris Lu
c45d6a865b
Merge branch 'master' into support_ssd_volume
2020-12-13 22:14:18 -08:00
Chris Lu
38fc89d041
filer store: sql update if any insert error happens
...
fix https://github.com/chrislusf/seaweedfs/issues/1673
2020-12-13 20:49:44 -08:00
Chris Lu
23280257df
sql put kv: avoid unnecessary update
2020-12-13 19:45:47 -08:00
Chris Lu
a9db24cd05
master allocate volumes if ssd type runs out
2020-12-13 19:44:57 -08:00
Chris Lu
ac4d4a65af
Update Makefile
2020-12-13 12:34:58 -08:00
Chris Lu
8baba93fce
rename parameter name to "disk"
2020-12-13 12:06:21 -08:00
Chris Lu
51eadaf2b6
rename parameter name to "disk"
2020-12-13 12:05:31 -08:00
Chris Lu
0d2ec832e2
rename from volumeType to diskType
2020-12-13 11:59:32 -08:00
Chris Lu
715b199eeb
fix tests
2020-12-13 04:14:50 -08:00
Chris Lu
66cd219b15
Merge branch 'master' into support_ssd_volume
2020-12-13 04:08:36 -08:00
Chris Lu
813453f3e1
2.15
2020-12-13 03:45:35 -08:00
Chris Lu
17372ac3da
balance ssd volumes
2020-12-13 03:40:33 -08:00
Chris Lu
d156c74ec0
volume server set volume type and heartbeat to the master
2020-12-13 03:11:24 -08:00
Chris Lu
e9cd798bd3
adding volume type
2020-12-13 00:58:58 -08:00
Chris Lu
16cd6fb278
fix variable name
2020-12-12 21:19:40 -08:00
Chris Lu
5c465293e9
correctly determine whether a folder is empty
...
avoid edge cases that deleting the folder if previous 32 directories are all empty
early terminate if one file is found
2020-12-12 16:19:29 -08:00
Chris Lu
14910d035c
the pagination size was too big for recursive deletion
2020-12-12 13:26:10 -08:00
Chris Lu
f930c713fc
more efficient recursion
2020-12-12 13:25:19 -08:00
Chris Lu
316d1b4e69
refactor APIs
2020-12-12 12:42:53 -08:00
Chris Lu
03637d6f57
s3: move "delete-directory-if-empty" to read time
...
move "delete-directory-if-empty" to read time instead of entry deletion time
the listing speed for a s3 bucket folder will slow down if it has many sub folders
related to 0d345ac97d
fix https://github.com/chrislusf/seaweedfs/issues/1647
fix https://github.com/chrislusf/seaweedfs/issues/1670
2020-12-12 03:38:34 -08:00
Chris Lu
37075a414d
adjust logs
2020-12-12 03:33:57 -08:00
Chris Lu
ae655033ac
adjust logging
2020-12-11 16:57:53 -08:00
Chris Lu
d986c7196d
use append time instead of filer's own modification time
...
fix https://github.com/chrislusf/seaweedfs/issues/1669
2020-12-11 16:55:18 -08:00
Chris Lu
3fedfec1e7
check cross device rename error
2020-12-10 23:50:32 -08:00
Chris Lu
c2f18a10cb
minor
2020-12-10 22:23:22 -08:00
Chris Lu
4b0c2a846b
add logs
2020-12-10 19:55:28 -08:00
Chris Lu
83078ac6ce
filer: change to /etc/seaweedfs folder on filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1666
2020-12-10 11:11:02 -08:00
Chris Lu
26731694f8
s3: use static configuration by default
...
So that users can still use the previous configuration files.
If leave it empty, s3 will try to use the version from filer
2020-12-10 00:59:04 -08:00
Chris Lu
765b3ef1be
save /etc/iam/identity.json inside filer store
2020-12-10 00:15:22 -08:00
Chris Lu
97c942b5ee
paginate through large directories
2020-12-09 23:23:38 -08:00
Chris Lu
02a160c3fd
s3: recursively iterate all sub folders
...
fix https://github.com/chrislusf/seaweedfs/issues/1656
2020-12-09 16:47:34 -08:00
Chris Lu
a9c619c270
added error handling for saving
2020-12-09 01:34:24 -08:00
Chris Lu
4211601eab
set file handle reader to nil
2020-12-08 22:48:18 -08:00
Chris Lu
b52ae9cef8
add s3 debug
2020-12-08 22:27:04 -08:00
Chris Lu
8e78187a97
add back last read chunk cache to reader and properly close the reader
2020-12-08 22:26:46 -08:00
Chris Lu
900d22c6ec
mount: avoid memory leaking read buffer
...
fix https://github.com/chrislusf/seaweedfs/issues/1654
the reader goes together with the file handle, which may stay for a long time.
2020-12-08 02:38:53 -08:00
Chris Lu
06bb7bf6c0
adding /usr/local/etc/seaweedfs for bsd style config search path
...
related to https://reviews.freebsd.org/D27391
2020-12-07 16:46:48 -08:00
Chris Lu
fce8803087
break import cycle
2020-12-07 00:29:17 -08:00
Chris Lu
606051cc79
filer: cache "/etc" filder small files to filer store
2020-12-07 00:11:35 -08:00
Chris Lu
ac22f1dd26
filer subscribe: handle rename subscription
2020-12-07 00:10:49 -08:00
Chris Lu
eed87791b7
s3: subscribe to s3.configure changes
2020-12-07 00:10:29 -08:00
Chris Lu
01e2da5782
refactoring
2020-12-06 23:16:20 -08:00
Chris Lu
77286f8bea
fix test
2020-12-06 21:56:13 -08:00
Chris Lu
57578a6cd1
change parameter help message
2020-12-06 21:54:55 -08:00
Chris Lu
78f3ab439c
refactoring
2020-12-06 21:54:34 -08:00
Chris Lu
9acda432fe
fix import cycle
2020-12-06 20:12:52 -08:00
Chris Lu
ae5eb85a06
refactoring
2020-12-06 20:05:06 -08:00
Chris Lu
5931a2f53f
filer conf: support json conf only
2020-12-06 19:47:06 -08:00
Chris Lu
4c72482536
shell: fix fs.configure
2020-12-06 19:43:31 -08:00
Chris Lu
46b91228af
2.14
2020-12-06 18:56:58 -08:00
Chris Lu
263eb29e9f
filer: add option to cache small files to filer store
2020-12-04 22:39:43 -08:00
Chris Lu
d171d9f988
volume: ensure the volume dat and idx files are always closed
...
fix https://github.com/chrislusf/seaweedfs/issues/1646
2020-12-04 21:50:26 -08:00
Konstantin Lebedev
eadbba5c95
security master use access.ui
2020-12-04 12:57:59 +05:00
Chris Lu
cc839f935d
better unit detection
2020-12-03 19:23:59 -08:00
Chris Lu
b219ccfe68
Update volume_ttl_test.go
2020-12-03 12:55:44 -08:00
Chris Lu
a83b8e7b01
ttl fix: 3y was converted into 96m
2020-12-03 10:53:29 -08:00
Chris Lu
5fb60b7135
Merge pull request #1596 from kmlebedev/store_s3cred
...
S3 credentials store in filer
2020-12-03 00:40:20 -08:00
Konstantin Lebedev
14699dfcef
use content field of entry
2020-12-02 17:19:05 +05:00
Konstantin Lebedev
a3d4b50a49
use entry content filed
2020-12-02 16:12:13 +05:00
Chris Lu
003b6245e7
fix nil
2020-12-02 00:09:19 -08:00
Chris Lu
f3bb645018
file open error
2020-12-01 23:37:49 -08:00
Chris Lu
dc0bc48257
return file open error
2020-12-01 23:36:49 -08:00
Chris Lu
45dba088cd
fix tests
2020-12-01 19:37:21 -08:00
Chris Lu
04062c56c7
webdav: improve webdav upload speed
2020-12-01 15:32:27 -08:00
Konstantin Lebedev
03620776ec
Merge branch 'upstream_master' into store_s3cred
2020-12-01 16:03:34 +05:00
Chris Lu
005a6123e9
Merge pull request #1643 from hilimd/master
...
fix tls grpc ca path
2020-11-30 17:16:46 -08:00
Chris Lu
f4abd01adf
filer: cache small file to filer store
2020-11-30 04:34:04 -08:00
Chris Lu
a9c6be5fc3
set creation time
2020-11-30 03:11:52 -08:00
Chris Lu
141ce67c09
close http request body
2020-11-30 02:45:00 -08:00
Chris Lu
0e99531dbf
2.13
2020-11-29 17:00:03 -08:00
Chris Lu
965413c21b
shell: add volume.vacuum command
2020-11-28 23:18:02 -08:00
Chris Lu
96c48bc8a8
fix test
2020-11-28 03:03:43 -08:00
Chris Lu
c57a7839ec
adjust comments
2020-11-28 03:01:03 -08:00
Chris Lu
8438fb0cc3
fix ec shard spreading error
2020-11-28 02:21:16 -08:00
Chris Lu
536202e4e0
minor
2020-11-28 02:16:06 -08:00
Chris Lu
99c4e50d3d
minor
2020-11-28 00:14:11 -08:00
Chris Lu
2c913dde04
volume: detect and drop volumes with disk IO error
...
from Jethro in slack:
is it possible to make the assign request a bit smarter? Currently I’m in the state that a disk failed but all assign request are being send to this volume. It would be cool if the master sees this and stopped using this volume.
e=HTTP(http://x:8089/913,045a782b63176edf ) not 200 but 500 Internal Server Error
Body={"size":740167,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"ee4381e202212ff3aee647704c036689"}
e=HTTP(http://x:8089/913,045a782c90240077 ) not 200 but 500 Internal Server Error
Body={"size":792779,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"c43463ccc11eb6eb2fc306f407a6a953"}
e=HTTP(http://x:8089/913,045a782e6b7901ea ) not 200 but 500 Internal Server Error
Body={"size":3962392,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"04c91198e9b276c81f11dbf189af5d28"}
2020-11-28 00:09:29 -08:00
Chris Lu
9ac4935f22
read from volume index file directly instead of open a separate file
...
fix https://github.com/chrislusf/seaweedfs/issues/1640
read from volume index file directly instead of open a separate file,
to ensure reading latest index entries.
2020-11-27 16:18:48 -08:00
Chris Lu
85554bea38
filer: readonly handle static resources
...
fix https://github.com/chrislusf/seaweedfs/issues/1641
2020-11-27 13:42:14 -08:00
Chris Lu
6d30b21b10
volume: add "-dir.idx" option for separate index storage
...
fix https://github.com/chrislusf/seaweedfs/issues/1265
2020-11-27 03:17:10 -08:00
Chris Lu
3c229eb677
refactor
2020-11-26 17:21:55 -08:00
Chris Lu
c0fb4a3766
add comments
2020-11-26 15:22:42 -08:00
Chris Lu
6783ae5f8f
rename
2020-11-26 15:19:43 -08:00
Chris Lu
983fd90010
marked as WIP
2020-11-26 15:08:52 -08:00
Chris Lu
97f706f35b
just refactoring
2020-11-26 15:08:05 -08:00
Chris Lu
b11449f955
pass in dir.idx parameter to DiskLocation
2020-11-26 14:59:03 -08:00
Chris Lu
0d345ac97d
s3: remove empty parent folder on delete
...
fix https://github.com/chrislusf/seaweedfs/issues/1637
2020-11-26 12:21:58 -08:00
Chris Lu
cc2bd97ad9
refactor
2020-11-26 11:25:56 -08:00
Chris Lu
0da7ecfd29
go fmt
2020-11-26 11:22:30 -08:00
Chris Lu
effa00ed08
refactor
2020-11-26 11:21:58 -08:00
Chris Lu
513bcd6e0d
filer: avoid duplicated FindEntry for deletion
2020-11-26 11:14:56 -08:00
Chris Lu
1ae108efca
add debug message
2020-11-26 10:41:10 -08:00
Chris Lu
a102157020
do not retry if failed with a non transport related error
2020-11-26 01:29:30 -08:00
Konstantin Lebedev
4e55baf5b1
s3 config changes
2020-11-26 03:50:53 +05:00
Konstantin Lebedev
52c8f2fc9a
s3iam test
2020-11-26 02:26:45 +05:00
Konstantin Lebedev
a26f1b2040
new pkg s3iam
2020-11-26 01:30:11 +05:00
Konstantin Lebedev
6206737df2
s3 configure
2020-11-25 21:02:31 +05:00
Chris Lu
eab53ea80d
filer leveldb store: a bit more efficient directory listing with prefix
2020-11-22 21:10:41 -08:00
Chris Lu
d3be2ceedd
2.12
2020-11-22 17:17:33 -08:00
Chris Lu
c7ebadc25d
avoid possible concurrent access inside ensureCorrectWritables()
2020-11-22 17:15:59 -08:00
hilimd
9aa990f80f
Merge pull request #39 from chrislusf/master
...
sync
2020-11-22 20:29:17 +08:00
limd
e8296104fc
fix tls grpc ca path
2020-11-22 20:27:15 +08:00
Chris Lu
92f906b6fc
remove zstd
...
fix https://github.com/chrislusf/seaweedfs/issues/1629
2020-11-21 13:06:45 -08:00
Aleksandr Demshin
176950ea78
fix typo
2020-11-20 19:50:46 +05:00
Chris Lu
7385103057
fix volume placement validating
...
fix https://github.com/chrislusf/seaweedfs/issues/1626
2020-11-20 01:12:25 -08:00
Konstantin Lebedev
27e73de797
Merge branch 'upstream_master' into store_s3cred
...
# Conflicts:
# weed/s3api/filer_util.go
2020-11-19 18:16:44 +05:00
Chris Lu
290b5e2cd0
directly delete file chunks
...
keeping current async deletions for now
2020-11-17 17:20:21 -08:00
Chris Lu
781585b195
remove unnecessary checking
2020-11-17 17:00:06 -08:00
Chris Lu
8cb8cd4cc5
add locks
2020-11-17 16:59:48 -08:00
Chris Lu
7c9e592c2d
s3: avoid nil resp when having error
...
fix https://github.com/chrislusf/seaweedfs/issues/1622
2020-11-17 11:23:17 -08:00
Chris Lu
dc304342b2
fs.configure: configurable volume growth
2020-11-17 01:00:02 -08:00
Chris Lu
b04375a3d1
use alternative assign request
2020-11-17 00:36:21 -08:00
Chris Lu
9add554feb
fail fast if user configures collection for a bucket
2020-11-16 19:57:08 -08:00
Chris Lu
6204dc6e85
add comment
2020-11-16 16:57:31 -08:00
Chris Lu
ef3e068c51
deprecating "filer.options.buckets_fsync" in filer.toml
2020-11-16 16:56:49 -08:00
Chris Lu
5f19e81dab
filer confi: support hierachical configuration
2020-11-16 16:50:12 -08:00
Chris Lu
ed1ce3f299
adjust help message
2020-11-16 09:54:26 -08:00
Chris Lu
c0d279c54e
filere.conf: prefer to use json format
2020-11-15 21:48:17 -08:00
Chris Lu
83527a8f55
adds more help message
2020-11-15 21:31:14 -08:00
Chris Lu
af658ea970
fix typo
2020-11-15 21:16:28 -08:00
Chris Lu
98827d68d9
2.11
2020-11-15 20:25:56 -08:00
Chris Lu
ee2fa14dbe
filer conf: delete location specific configuration
2020-11-15 20:15:47 -08:00
Chris Lu
2bd6fd3bbe
remove unused function
2020-11-15 20:15:07 -08:00
Chris Lu
71056dae07
fs.configure: read and local add filer configuration
2020-11-15 18:09:35 -08:00
Chris Lu
0ea5c087ce
go fmt
2020-11-15 16:59:28 -08:00
Chris Lu
95c0de285d
refactoring
2020-11-15 16:58:48 -08:00
Chris Lu
500bcab953
refactoring
2020-11-15 14:41:56 -08:00
Chris Lu
590f02179d
filer: load filer conf when starting
2020-11-15 14:06:03 -08:00
Chris Lu
0a406f652e
load filer conf and match by prefix
2020-11-15 00:26:05 -08:00
Chris Lu
68043cfcac
add reference implementation to detect create/update/delete/rename events
2020-11-14 21:21:58 -08:00
Chris Lu
0fc9ffc603
able to read filer.conf if updated
2020-11-14 21:21:20 -08:00
Chris Lu
442e092995
filer watch "/etc" folder for configuration changes
2020-11-14 14:26:08 -08:00
ruitao.liu
a9990a1dc6
adjust check bucket if exist or has access.
2020-11-13 17:13:20 +08:00
Chris Lu
e6333da65a
enable admin to access all buckets
2020-11-12 13:57:54 -08:00
Chris Lu
7094492428
fix ErrNotFound
2020-11-12 13:30:46 -08:00
Chris Lu
c6a0704019
rename function
2020-11-12 13:30:08 -08:00
Chris Lu
ed3b0f81c6
Merge pull request #1613 from taozix/master
...
check permission for bucket delete/head.
2020-11-12 13:25:28 -08:00
ruitao.liu
c4f0fd6e1b
skip if entry.Extended map is nil.
2020-11-12 17:59:31 +08:00
ruitao.liu
ab966410d2
return NoSuchBucket instead of InternalError delete non-existed bucket.
2020-11-12 16:44:16 +08:00
Chris Lu
a4dfa00030
adjust filer conf
2020-11-12 00:39:59 -08:00
Chris Lu
2251a8e472
minor
2020-11-12 00:38:58 -08:00
Chris Lu
5a16f17e47
remove unused message type
2020-11-12 00:38:23 -08:00
ruitao.liu
e06676f007
check permission for bucket delete/head.
2020-11-12 16:15:59 +08:00
Chris Lu
c32b6f4d9d
prepare for path based configuration
2020-11-11 23:46:45 -08:00
Chris Lu
0fbc27b640
Merge pull request #1612 from taozix/master
...
add owner attr for bucket.
2020-11-11 22:19:47 -08:00
ruitao.liu
d7cc0498e0
check if bucket already exists.
2020-11-12 14:11:03 +08:00
Chris Lu
827930e249
Merge pull request #1610 from kmlebedev/prefer_read_in_this_dc
...
prefer to read from volumes in this data center
2020-11-11 22:09:41 -08:00
Konstantin Lebedev
0880aff224
update dc flag desc
2020-11-12 10:07:52 +05:00
ruitao.liu
dbba8cb57e
fix if nil map in entry.
2020-11-12 11:50:19 +08:00
Konstantin Lebedev
aa30604cec
fix tests
2020-11-12 08:49:38 +05:00
Konstantin Lebedev
1eec5c8d5d
gen pb
2020-11-12 04:10:06 +05:00
Konstantin Lebedev
fc7baef5bb
fiil serverUrls sorted by data center
2020-11-12 02:13:33 +05:00
Chris Lu
745ee8d8f3
avoid unnecessary error message
...
fix https://github.com/chrislusf/seaweedfs/issues/1611
2020-11-11 13:00:05 -08:00
Chris Lu
a29abd8600
add a {X-,}Idempotency-Key header for http POST
...
fix https://github.com/chrislusf/seaweedfs/issues/1592
according to https://github.com/golang/go/issues/19943
2020-11-11 12:56:53 -08:00
Chris Lu
8708e7a64d
print out data size
2020-11-11 12:51:44 -08:00
Chris Lu
c6bd244ebd
add TODO
2020-11-11 12:51:27 -08:00
Konstantin Lebedev
b73ef6aa95
Merge remote-tracking branch 'upstream/master' into prefer_read_in_this_dc
2020-11-11 23:29:57 +05:00
Konstantin Lebedev
dc26012a3b
initial
2020-11-11 15:03:47 +05:00
Chris Lu
86cdb2a3e5
adjust logs
2020-11-11 02:01:24 -08:00
Chris Lu
73f934d5de
s3: do not close reader too early
...
fix https://github.com/chrislusf/seaweedfs/issues/1609
2020-11-11 01:49:39 -08:00
ruitao.liu
5b636b3242
Add bucket owner attr.
2020-11-11 16:20:59 +08:00
Chris Lu
15956d8a8e
2.10
2020-11-10 23:17:13 -08:00
Chris Lu
de3bdd0651
delete old volume replica
...
related to https://github.com/chrislusf/seaweedfs/issues/1607
old is:
* older compaction revision
* older modified time
* smaller volume size
2020-11-10 12:26:05 -08:00
Chris Lu
1dd3a6ac36
avoid verbose logs
2020-11-09 15:56:11 -08:00
Chris Lu
6856b0d57e
filer: add API to add/modify/delete tagging
2020-11-09 01:00:07 -08:00
Konstantin Lebedev
6cd6ff6962
avoid blank response for 404
2020-11-04 22:54:47 +05:00
Chris Lu
0dafcf1f5a
volume: detect max volume count changes based on disk usage
...
fix https://github.com/chrislusf/seaweedfs/issues/1594
2020-11-03 14:43:17 -08:00
Chris Lu
06ee199ef3
ensure entryViewCache is in sync with the chunks
2020-11-03 12:22:19 -08:00
Chris Lu
b132cb893f
in case some chunks modified earliar are added later
2020-11-03 12:22:19 -08:00
Chris Lu
0ae4d7bfdf
lock "addChunks()"
2020-11-03 12:22:19 -08:00
Konstantin Lebedev
e1190b3224
load S3 config from filer
...
https://github.com/chrislusf/seaweedfs/issues/1500
2020-11-03 21:45:56 +05:00
Chris Lu
d29b787220
store together with "Seaweed-" prefix
2020-11-03 00:21:10 -08:00
Chris Lu
5448781dfc
filer: Headers with "Seaweed-" prefix are stored as extended properties
2020-11-03 00:15:51 -08:00
Chris Lu
0ecff9521d
avoid []byte reuse
2020-11-02 23:27:54 -08:00
Chris Lu
031aa70da6
Revert "revert bytebufferpool, seems problematic"
...
This reverts commit e0fcab47c0
.
2020-11-02 19:11:25 -08:00
Chris Lu
7d5db73d98
go fmt
2020-11-02 14:21:07 -08:00
Chris Lu
a9662d44ac
avoid closing chunkSaveErrChan too early
2020-11-02 14:20:38 -08:00
Chris Lu
e0fcab47c0
revert bytebufferpool, seems problematic
2020-11-02 13:49:38 -08:00
Chris Lu
e1423a83dd
mount: clear possible leftover entry data in memory
2020-11-01 23:28:56 -08:00
Chris Lu
9b10d1ac33
2.08
2020-11-01 12:45:20 -08:00
Chris Lu
8750cac090
move to util.RetryWaitTime
2020-11-01 02:36:43 -08:00
Chris Lu
ef908e166b
break import cycle
2020-11-01 02:28:11 -08:00
Chris Lu
4609644658
mount: remove files that may still are open
...
related to https://github.com/chrislusf/seaweedfs/issues/1581
2020-11-01 01:33:26 -08:00
Chris Lu
df8d976bb0
refactoring
2020-11-01 01:58:48 -07:00
Chris Lu
7192a378cc
add retry to master clients also
2020-11-01 01:40:16 -07:00
Chris Lu
004aa9ca5d
only retry transport related errors
2020-11-01 01:21:44 -07:00
Chris Lu
f7b21973ac
mount: add retry for all operations with filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1589
2020-11-01 01:11:08 -07:00
Chris Lu
de86945aeb
go fmt
2020-10-31 16:45:38 -07:00
Chris Lu
306062b4e7
filer: add CORS support
2020-10-31 16:44:03 -07:00
Chris Lu
05acc33294
volume: add CORS support
2020-10-31 16:31:39 -07:00
Chris Lu
9708df47d1
ensure offset in the index file is positive
2020-10-31 15:50:01 -07:00
Chris Lu
10f9f1cc26
handle mknod requests
2020-10-30 23:51:32 -07:00
Chris Lu
8826601be1
mount: optional limit for the number of concurrent writers
2020-10-30 21:22:20 -07:00
Chris Lu
be95f68ca7
scaffold for sftpd
2020-10-30 20:57:08 -07:00
Konstantin Lebedev
d2a4ffdd60
glog
2020-10-31 03:21:54 +05:00
Konstantin Lebedev
46303c36bf
When the volume server is stopped, the master server immediately sees the deletion of volumes
2020-10-31 02:48:25 +05:00
Chris Lu
eb32af25d4
make space for the QR code
2020-10-30 10:55:31 -07:00
Chris Lu
7ecbb4b3c9
clean up
2020-10-30 10:46:31 -07:00
Kenny
9cc6575ab5
Update filer_copy.go
...
修改 weed filer.copy 上传目录时文件名变子目录的问题
2020-10-30 18:16:49 +08:00
Chris Lu
c057dd1f64
add back upload retries
2020-10-30 02:16:34 -07:00
Chris Lu
19098f2c2d
make a copy of the filer.entry for most read operations
2020-10-30 01:23:26 -07:00
Chris Lu
6135cbaa02
clear file.entry
2020-10-30 01:23:26 -07:00