Chris Lu
715b199eeb
fix tests
2020-12-13 04:14:50 -08:00
Chris Lu
66cd219b15
Merge branch 'master' into support_ssd_volume
2020-12-13 04:08:36 -08:00
Chris Lu
813453f3e1
2.15
2020-12-13 03:45:35 -08:00
Chris Lu
17372ac3da
balance ssd volumes
2020-12-13 03:40:33 -08:00
Chris Lu
d156c74ec0
volume server set volume type and heartbeat to the master
2020-12-13 03:11:24 -08:00
Chris Lu
e9cd798bd3
adding volume type
2020-12-13 00:58:58 -08:00
Chris Lu
16cd6fb278
fix variable name
2020-12-12 21:19:40 -08:00
Chris Lu
5c465293e9
correctly determine whether a folder is empty
...
avoid edge cases that deleting the folder if previous 32 directories are all empty
early terminate if one file is found
2020-12-12 16:19:29 -08:00
Chris Lu
14910d035c
the pagination size was too big for recursive deletion
2020-12-12 13:26:10 -08:00
Chris Lu
f930c713fc
more efficient recursion
2020-12-12 13:25:19 -08:00
Chris Lu
316d1b4e69
refactor APIs
2020-12-12 12:42:53 -08:00
Chris Lu
03637d6f57
s3: move "delete-directory-if-empty" to read time
...
move "delete-directory-if-empty" to read time instead of entry deletion time
the listing speed for a s3 bucket folder will slow down if it has many sub folders
related to 0d345ac97d
fix https://github.com/chrislusf/seaweedfs/issues/1647
fix https://github.com/chrislusf/seaweedfs/issues/1670
2020-12-12 03:38:34 -08:00
Chris Lu
37075a414d
adjust logs
2020-12-12 03:33:57 -08:00
Chris Lu
ae655033ac
adjust logging
2020-12-11 16:57:53 -08:00
Chris Lu
d986c7196d
use append time instead of filer's own modification time
...
fix https://github.com/chrislusf/seaweedfs/issues/1669
2020-12-11 16:55:18 -08:00
Chris Lu
3fedfec1e7
check cross device rename error
2020-12-10 23:50:32 -08:00
Chris Lu
c2f18a10cb
minor
2020-12-10 22:23:22 -08:00
Chris Lu
4b0c2a846b
add logs
2020-12-10 19:55:28 -08:00
Chris Lu
83078ac6ce
filer: change to /etc/seaweedfs folder on filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1666
2020-12-10 11:11:02 -08:00
Chris Lu
26731694f8
s3: use static configuration by default
...
So that users can still use the previous configuration files.
If leave it empty, s3 will try to use the version from filer
2020-12-10 00:59:04 -08:00
Chris Lu
765b3ef1be
save /etc/iam/identity.json inside filer store
2020-12-10 00:15:22 -08:00
Chris Lu
97c942b5ee
paginate through large directories
2020-12-09 23:23:38 -08:00
Chris Lu
02a160c3fd
s3: recursively iterate all sub folders
...
fix https://github.com/chrislusf/seaweedfs/issues/1656
2020-12-09 16:47:34 -08:00
Chris Lu
a9c619c270
added error handling for saving
2020-12-09 01:34:24 -08:00
Chris Lu
4211601eab
set file handle reader to nil
2020-12-08 22:48:18 -08:00
Chris Lu
b52ae9cef8
add s3 debug
2020-12-08 22:27:04 -08:00
Chris Lu
8e78187a97
add back last read chunk cache to reader and properly close the reader
2020-12-08 22:26:46 -08:00
Chris Lu
900d22c6ec
mount: avoid memory leaking read buffer
...
fix https://github.com/chrislusf/seaweedfs/issues/1654
the reader goes together with the file handle, which may stay for a long time.
2020-12-08 02:38:53 -08:00
Chris Lu
06bb7bf6c0
adding /usr/local/etc/seaweedfs for bsd style config search path
...
related to https://reviews.freebsd.org/D27391
2020-12-07 16:46:48 -08:00
Chris Lu
fce8803087
break import cycle
2020-12-07 00:29:17 -08:00
Chris Lu
606051cc79
filer: cache "/etc" filder small files to filer store
2020-12-07 00:11:35 -08:00
Chris Lu
ac22f1dd26
filer subscribe: handle rename subscription
2020-12-07 00:10:49 -08:00
Chris Lu
eed87791b7
s3: subscribe to s3.configure changes
2020-12-07 00:10:29 -08:00
Chris Lu
01e2da5782
refactoring
2020-12-06 23:16:20 -08:00
Chris Lu
77286f8bea
fix test
2020-12-06 21:56:13 -08:00
Chris Lu
57578a6cd1
change parameter help message
2020-12-06 21:54:55 -08:00
Chris Lu
78f3ab439c
refactoring
2020-12-06 21:54:34 -08:00
Chris Lu
9acda432fe
fix import cycle
2020-12-06 20:12:52 -08:00
Chris Lu
ae5eb85a06
refactoring
2020-12-06 20:05:06 -08:00
Chris Lu
5931a2f53f
filer conf: support json conf only
2020-12-06 19:47:06 -08:00
Chris Lu
4c72482536
shell: fix fs.configure
2020-12-06 19:43:31 -08:00
Chris Lu
46b91228af
2.14
2020-12-06 18:56:58 -08:00
Chris Lu
263eb29e9f
filer: add option to cache small files to filer store
2020-12-04 22:39:43 -08:00
Chris Lu
d171d9f988
volume: ensure the volume dat and idx files are always closed
...
fix https://github.com/chrislusf/seaweedfs/issues/1646
2020-12-04 21:50:26 -08:00
Konstantin Lebedev
eadbba5c95
security master use access.ui
2020-12-04 12:57:59 +05:00
Chris Lu
cc839f935d
better unit detection
2020-12-03 19:23:59 -08:00
Chris Lu
b219ccfe68
Update volume_ttl_test.go
2020-12-03 12:55:44 -08:00
Chris Lu
a83b8e7b01
ttl fix: 3y was converted into 96m
2020-12-03 10:53:29 -08:00
Chris Lu
5fb60b7135
Merge pull request #1596 from kmlebedev/store_s3cred
...
S3 credentials store in filer
2020-12-03 00:40:20 -08:00
Konstantin Lebedev
14699dfcef
use content field of entry
2020-12-02 17:19:05 +05:00
Konstantin Lebedev
a3d4b50a49
use entry content filed
2020-12-02 16:12:13 +05:00
Chris Lu
003b6245e7
fix nil
2020-12-02 00:09:19 -08:00
Chris Lu
f3bb645018
file open error
2020-12-01 23:37:49 -08:00
Chris Lu
dc0bc48257
return file open error
2020-12-01 23:36:49 -08:00
Chris Lu
45dba088cd
fix tests
2020-12-01 19:37:21 -08:00
Chris Lu
04062c56c7
webdav: improve webdav upload speed
2020-12-01 15:32:27 -08:00
Konstantin Lebedev
03620776ec
Merge branch 'upstream_master' into store_s3cred
2020-12-01 16:03:34 +05:00
Chris Lu
005a6123e9
Merge pull request #1643 from hilimd/master
...
fix tls grpc ca path
2020-11-30 17:16:46 -08:00
Chris Lu
f4abd01adf
filer: cache small file to filer store
2020-11-30 04:34:04 -08:00
Chris Lu
a9c6be5fc3
set creation time
2020-11-30 03:11:52 -08:00
Chris Lu
141ce67c09
close http request body
2020-11-30 02:45:00 -08:00
Chris Lu
0e99531dbf
2.13
2020-11-29 17:00:03 -08:00
Chris Lu
965413c21b
shell: add volume.vacuum command
2020-11-28 23:18:02 -08:00
Chris Lu
96c48bc8a8
fix test
2020-11-28 03:03:43 -08:00
Chris Lu
c57a7839ec
adjust comments
2020-11-28 03:01:03 -08:00
Chris Lu
8438fb0cc3
fix ec shard spreading error
2020-11-28 02:21:16 -08:00
Chris Lu
536202e4e0
minor
2020-11-28 02:16:06 -08:00
Chris Lu
99c4e50d3d
minor
2020-11-28 00:14:11 -08:00
Chris Lu
2c913dde04
volume: detect and drop volumes with disk IO error
...
from Jethro in slack:
is it possible to make the assign request a bit smarter? Currently I’m in the state that a disk failed but all assign request are being send to this volume. It would be cool if the master sees this and stopped using this volume.
e=HTTP(http://x:8089/913,045a782b63176edf ) not 200 but 500 Internal Server Error
Body={"size":740167,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"ee4381e202212ff3aee647704c036689"}
e=HTTP(http://x:8089/913,045a782c90240077 ) not 200 but 500 Internal Server Error
Body={"size":792779,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"c43463ccc11eb6eb2fc306f407a6a953"}
e=HTTP(http://x:8089/913,045a782e6b7901ea ) not 200 but 500 Internal Server Error
Body={"size":3962392,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"04c91198e9b276c81f11dbf189af5d28"}
2020-11-28 00:09:29 -08:00
Chris Lu
9ac4935f22
read from volume index file directly instead of open a separate file
...
fix https://github.com/chrislusf/seaweedfs/issues/1640
read from volume index file directly instead of open a separate file,
to ensure reading latest index entries.
2020-11-27 16:18:48 -08:00
Chris Lu
85554bea38
filer: readonly handle static resources
...
fix https://github.com/chrislusf/seaweedfs/issues/1641
2020-11-27 13:42:14 -08:00
Chris Lu
6d30b21b10
volume: add "-dir.idx" option for separate index storage
...
fix https://github.com/chrislusf/seaweedfs/issues/1265
2020-11-27 03:17:10 -08:00
Chris Lu
3c229eb677
refactor
2020-11-26 17:21:55 -08:00
Chris Lu
c0fb4a3766
add comments
2020-11-26 15:22:42 -08:00
Chris Lu
6783ae5f8f
rename
2020-11-26 15:19:43 -08:00
Chris Lu
983fd90010
marked as WIP
2020-11-26 15:08:52 -08:00
Chris Lu
97f706f35b
just refactoring
2020-11-26 15:08:05 -08:00
Chris Lu
b11449f955
pass in dir.idx parameter to DiskLocation
2020-11-26 14:59:03 -08:00
Chris Lu
0d345ac97d
s3: remove empty parent folder on delete
...
fix https://github.com/chrislusf/seaweedfs/issues/1637
2020-11-26 12:21:58 -08:00
Chris Lu
cc2bd97ad9
refactor
2020-11-26 11:25:56 -08:00
Chris Lu
0da7ecfd29
go fmt
2020-11-26 11:22:30 -08:00
Chris Lu
effa00ed08
refactor
2020-11-26 11:21:58 -08:00
Chris Lu
513bcd6e0d
filer: avoid duplicated FindEntry for deletion
2020-11-26 11:14:56 -08:00
Chris Lu
1ae108efca
add debug message
2020-11-26 10:41:10 -08:00
Chris Lu
a102157020
do not retry if failed with a non transport related error
2020-11-26 01:29:30 -08:00
Konstantin Lebedev
4e55baf5b1
s3 config changes
2020-11-26 03:50:53 +05:00
Konstantin Lebedev
52c8f2fc9a
s3iam test
2020-11-26 02:26:45 +05:00
Konstantin Lebedev
a26f1b2040
new pkg s3iam
2020-11-26 01:30:11 +05:00
Konstantin Lebedev
6206737df2
s3 configure
2020-11-25 21:02:31 +05:00
Chris Lu
eab53ea80d
filer leveldb store: a bit more efficient directory listing with prefix
2020-11-22 21:10:41 -08:00
Chris Lu
d3be2ceedd
2.12
2020-11-22 17:17:33 -08:00
Chris Lu
c7ebadc25d
avoid possible concurrent access inside ensureCorrectWritables()
2020-11-22 17:15:59 -08:00
hilimd
9aa990f80f
Merge pull request #39 from chrislusf/master
...
sync
2020-11-22 20:29:17 +08:00
limd
e8296104fc
fix tls grpc ca path
2020-11-22 20:27:15 +08:00
Chris Lu
92f906b6fc
remove zstd
...
fix https://github.com/chrislusf/seaweedfs/issues/1629
2020-11-21 13:06:45 -08:00
Aleksandr Demshin
176950ea78
fix typo
2020-11-20 19:50:46 +05:00
Chris Lu
7385103057
fix volume placement validating
...
fix https://github.com/chrislusf/seaweedfs/issues/1626
2020-11-20 01:12:25 -08:00
Konstantin Lebedev
27e73de797
Merge branch 'upstream_master' into store_s3cred
...
# Conflicts:
# weed/s3api/filer_util.go
2020-11-19 18:16:44 +05:00
Chris Lu
290b5e2cd0
directly delete file chunks
...
keeping current async deletions for now
2020-11-17 17:20:21 -08:00
Chris Lu
781585b195
remove unnecessary checking
2020-11-17 17:00:06 -08:00
Chris Lu
8cb8cd4cc5
add locks
2020-11-17 16:59:48 -08:00
Chris Lu
7c9e592c2d
s3: avoid nil resp when having error
...
fix https://github.com/chrislusf/seaweedfs/issues/1622
2020-11-17 11:23:17 -08:00
Chris Lu
dc304342b2
fs.configure: configurable volume growth
2020-11-17 01:00:02 -08:00
Chris Lu
b04375a3d1
use alternative assign request
2020-11-17 00:36:21 -08:00
Chris Lu
9add554feb
fail fast if user configures collection for a bucket
2020-11-16 19:57:08 -08:00
Chris Lu
6204dc6e85
add comment
2020-11-16 16:57:31 -08:00
Chris Lu
ef3e068c51
deprecating "filer.options.buckets_fsync" in filer.toml
2020-11-16 16:56:49 -08:00
Chris Lu
5f19e81dab
filer confi: support hierachical configuration
2020-11-16 16:50:12 -08:00
Chris Lu
ed1ce3f299
adjust help message
2020-11-16 09:54:26 -08:00
Chris Lu
c0d279c54e
filere.conf: prefer to use json format
2020-11-15 21:48:17 -08:00
Chris Lu
83527a8f55
adds more help message
2020-11-15 21:31:14 -08:00
Chris Lu
af658ea970
fix typo
2020-11-15 21:16:28 -08:00
Chris Lu
98827d68d9
2.11
2020-11-15 20:25:56 -08:00
Chris Lu
ee2fa14dbe
filer conf: delete location specific configuration
2020-11-15 20:15:47 -08:00
Chris Lu
2bd6fd3bbe
remove unused function
2020-11-15 20:15:07 -08:00
Chris Lu
71056dae07
fs.configure: read and local add filer configuration
2020-11-15 18:09:35 -08:00
Chris Lu
0ea5c087ce
go fmt
2020-11-15 16:59:28 -08:00
Chris Lu
95c0de285d
refactoring
2020-11-15 16:58:48 -08:00
Chris Lu
500bcab953
refactoring
2020-11-15 14:41:56 -08:00
Chris Lu
590f02179d
filer: load filer conf when starting
2020-11-15 14:06:03 -08:00
Chris Lu
0a406f652e
load filer conf and match by prefix
2020-11-15 00:26:05 -08:00
Chris Lu
68043cfcac
add reference implementation to detect create/update/delete/rename events
2020-11-14 21:21:58 -08:00
Chris Lu
0fc9ffc603
able to read filer.conf if updated
2020-11-14 21:21:20 -08:00
Chris Lu
442e092995
filer watch "/etc" folder for configuration changes
2020-11-14 14:26:08 -08:00
ruitao.liu
a9990a1dc6
adjust check bucket if exist or has access.
2020-11-13 17:13:20 +08:00
Chris Lu
e6333da65a
enable admin to access all buckets
2020-11-12 13:57:54 -08:00
Chris Lu
7094492428
fix ErrNotFound
2020-11-12 13:30:46 -08:00
Chris Lu
c6a0704019
rename function
2020-11-12 13:30:08 -08:00
Chris Lu
ed3b0f81c6
Merge pull request #1613 from taozix/master
...
check permission for bucket delete/head.
2020-11-12 13:25:28 -08:00
ruitao.liu
c4f0fd6e1b
skip if entry.Extended map is nil.
2020-11-12 17:59:31 +08:00
ruitao.liu
ab966410d2
return NoSuchBucket instead of InternalError delete non-existed bucket.
2020-11-12 16:44:16 +08:00
Chris Lu
a4dfa00030
adjust filer conf
2020-11-12 00:39:59 -08:00
Chris Lu
2251a8e472
minor
2020-11-12 00:38:58 -08:00
Chris Lu
5a16f17e47
remove unused message type
2020-11-12 00:38:23 -08:00
ruitao.liu
e06676f007
check permission for bucket delete/head.
2020-11-12 16:15:59 +08:00
Chris Lu
c32b6f4d9d
prepare for path based configuration
2020-11-11 23:46:45 -08:00
Chris Lu
0fbc27b640
Merge pull request #1612 from taozix/master
...
add owner attr for bucket.
2020-11-11 22:19:47 -08:00
ruitao.liu
d7cc0498e0
check if bucket already exists.
2020-11-12 14:11:03 +08:00
Chris Lu
827930e249
Merge pull request #1610 from kmlebedev/prefer_read_in_this_dc
...
prefer to read from volumes in this data center
2020-11-11 22:09:41 -08:00
Konstantin Lebedev
0880aff224
update dc flag desc
2020-11-12 10:07:52 +05:00
ruitao.liu
dbba8cb57e
fix if nil map in entry.
2020-11-12 11:50:19 +08:00
Konstantin Lebedev
aa30604cec
fix tests
2020-11-12 08:49:38 +05:00
Konstantin Lebedev
1eec5c8d5d
gen pb
2020-11-12 04:10:06 +05:00
Konstantin Lebedev
fc7baef5bb
fiil serverUrls sorted by data center
2020-11-12 02:13:33 +05:00
Chris Lu
745ee8d8f3
avoid unnecessary error message
...
fix https://github.com/chrislusf/seaweedfs/issues/1611
2020-11-11 13:00:05 -08:00
Chris Lu
a29abd8600
add a {X-,}Idempotency-Key header for http POST
...
fix https://github.com/chrislusf/seaweedfs/issues/1592
according to https://github.com/golang/go/issues/19943
2020-11-11 12:56:53 -08:00
Chris Lu
8708e7a64d
print out data size
2020-11-11 12:51:44 -08:00
Chris Lu
c6bd244ebd
add TODO
2020-11-11 12:51:27 -08:00
Konstantin Lebedev
b73ef6aa95
Merge remote-tracking branch 'upstream/master' into prefer_read_in_this_dc
2020-11-11 23:29:57 +05:00
Konstantin Lebedev
dc26012a3b
initial
2020-11-11 15:03:47 +05:00
Chris Lu
86cdb2a3e5
adjust logs
2020-11-11 02:01:24 -08:00
Chris Lu
73f934d5de
s3: do not close reader too early
...
fix https://github.com/chrislusf/seaweedfs/issues/1609
2020-11-11 01:49:39 -08:00
ruitao.liu
5b636b3242
Add bucket owner attr.
2020-11-11 16:20:59 +08:00
Chris Lu
15956d8a8e
2.10
2020-11-10 23:17:13 -08:00
Chris Lu
de3bdd0651
delete old volume replica
...
related to https://github.com/chrislusf/seaweedfs/issues/1607
old is:
* older compaction revision
* older modified time
* smaller volume size
2020-11-10 12:26:05 -08:00
Chris Lu
1dd3a6ac36
avoid verbose logs
2020-11-09 15:56:11 -08:00
Chris Lu
6856b0d57e
filer: add API to add/modify/delete tagging
2020-11-09 01:00:07 -08:00
Konstantin Lebedev
6cd6ff6962
avoid blank response for 404
2020-11-04 22:54:47 +05:00
Chris Lu
0dafcf1f5a
volume: detect max volume count changes based on disk usage
...
fix https://github.com/chrislusf/seaweedfs/issues/1594
2020-11-03 14:43:17 -08:00
Chris Lu
06ee199ef3
ensure entryViewCache is in sync with the chunks
2020-11-03 12:22:19 -08:00
Chris Lu
b132cb893f
in case some chunks modified earliar are added later
2020-11-03 12:22:19 -08:00
Chris Lu
0ae4d7bfdf
lock "addChunks()"
2020-11-03 12:22:19 -08:00
Konstantin Lebedev
e1190b3224
load S3 config from filer
...
https://github.com/chrislusf/seaweedfs/issues/1500
2020-11-03 21:45:56 +05:00
Chris Lu
d29b787220
store together with "Seaweed-" prefix
2020-11-03 00:21:10 -08:00
Chris Lu
5448781dfc
filer: Headers with "Seaweed-" prefix are stored as extended properties
2020-11-03 00:15:51 -08:00
Chris Lu
0ecff9521d
avoid []byte reuse
2020-11-02 23:27:54 -08:00
Chris Lu
031aa70da6
Revert "revert bytebufferpool, seems problematic"
...
This reverts commit e0fcab47c0
.
2020-11-02 19:11:25 -08:00
Chris Lu
7d5db73d98
go fmt
2020-11-02 14:21:07 -08:00
Chris Lu
a9662d44ac
avoid closing chunkSaveErrChan too early
2020-11-02 14:20:38 -08:00
Chris Lu
e0fcab47c0
revert bytebufferpool, seems problematic
2020-11-02 13:49:38 -08:00
Chris Lu
e1423a83dd
mount: clear possible leftover entry data in memory
2020-11-01 23:28:56 -08:00
Chris Lu
9b10d1ac33
2.08
2020-11-01 12:45:20 -08:00
Chris Lu
8750cac090
move to util.RetryWaitTime
2020-11-01 02:36:43 -08:00
Chris Lu
ef908e166b
break import cycle
2020-11-01 02:28:11 -08:00
Chris Lu
4609644658
mount: remove files that may still are open
...
related to https://github.com/chrislusf/seaweedfs/issues/1581
2020-11-01 01:33:26 -08:00
Chris Lu
df8d976bb0
refactoring
2020-11-01 01:58:48 -07:00
Chris Lu
7192a378cc
add retry to master clients also
2020-11-01 01:40:16 -07:00
Chris Lu
004aa9ca5d
only retry transport related errors
2020-11-01 01:21:44 -07:00
Chris Lu
f7b21973ac
mount: add retry for all operations with filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1589
2020-11-01 01:11:08 -07:00
Chris Lu
de86945aeb
go fmt
2020-10-31 16:45:38 -07:00
Chris Lu
306062b4e7
filer: add CORS support
2020-10-31 16:44:03 -07:00
Chris Lu
05acc33294
volume: add CORS support
2020-10-31 16:31:39 -07:00
Chris Lu
9708df47d1
ensure offset in the index file is positive
2020-10-31 15:50:01 -07:00
Chris Lu
10f9f1cc26
handle mknod requests
2020-10-30 23:51:32 -07:00
Chris Lu
8826601be1
mount: optional limit for the number of concurrent writers
2020-10-30 21:22:20 -07:00
Chris Lu
be95f68ca7
scaffold for sftpd
2020-10-30 20:57:08 -07:00
Konstantin Lebedev
d2a4ffdd60
glog
2020-10-31 03:21:54 +05:00
Konstantin Lebedev
46303c36bf
When the volume server is stopped, the master server immediately sees the deletion of volumes
2020-10-31 02:48:25 +05:00
Chris Lu
eb32af25d4
make space for the QR code
2020-10-30 10:55:31 -07:00
Chris Lu
7ecbb4b3c9
clean up
2020-10-30 10:46:31 -07:00
Kenny
9cc6575ab5
Update filer_copy.go
...
修改 weed filer.copy 上传目录时文件名变子目录的问题
2020-10-30 18:16:49 +08:00
Chris Lu
c057dd1f64
add back upload retries
2020-10-30 02:16:34 -07:00
Chris Lu
19098f2c2d
make a copy of the filer.entry for most read operations
2020-10-30 01:23:26 -07:00
Chris Lu
6135cbaa02
clear file.entry
2020-10-30 01:23:26 -07:00
Chris Lu
02dc51b1ed
properly clear out the file.entry
2020-10-29 23:31:06 -07:00
Chris Lu
6560ac6466
volume loading: trim out unreachable idx file content
...
fix https://github.com/chrislusf/seaweedfs/issues/1583
2020-10-29 22:25:23 -07:00
Chris Lu
0b68b68ec4
reload entry only when it is a hard link
...
fix https://github.com/chrislusf/seaweedfs/issues/1581
2020-10-29 16:32:45 -07:00
Chris Lu
47047516d6
Revert "increase default volume file size limit to 1024"
...
This reverts commit 0983060a
I am confused why this was changed to 1024 by myself. 1GB should be too large for most cases.
2020-10-29 15:46:29 -07:00
Chris Lu
6f8b426f4f
Merge pull request #1578 from taozix/master
...
S3 bucket list, response with uploaded storageclass.
2020-10-29 08:32:27 -07:00
ruitao.liu
22a9ea0512
adjust s3 header file.
2020-10-29 16:05:40 +08:00
Chris Lu
c6d9974fe7
server: Add option to enable/disable volume server
...
fix https://github.com/chrislusf/seaweedfs/issues/1572
2020-10-29 00:24:18 -07:00
ruitao.liu
b917be7955
S3 bucket list, response with uploaded storageclass.
2020-10-29 14:57:19 +08:00
Konstantin Lebedev
918546bbdb
add WaitForReady to grpc DialOption
2020-10-29 08:29:04 +05:00
Konstantin Lebedev
fab01f9d8d
add readonly
2020-10-28 23:17:03 +05:00
Konstantin Lebedev
884db215a1
add shell command volume mark writable
2020-10-28 22:47:09 +05:00
Konstantin Lebedev
1ad1b8c4f6
collection Volume ReadOnly Count with detailed status
2020-10-28 17:38:26 +05:00
Chris Lu
326fcdd86b
Merge pull request #1569 from taozix/master
...
save s3 metadata to filer.
2020-10-28 03:27:24 -07:00
ruitao.liu
2dcc178d0a
save s3 metadata to filer.
2020-10-28 18:16:05 +08:00
Chris Lu
744c5594da
volume: automatically trim out unreachable entries
2020-10-28 01:14:39 -07:00
Chris Lu
3e925faddd
trim out name extension
2020-10-28 01:14:05 -07:00
shibinbin
a67eb1afa1
fix: restart volumeserver sometime occurred out of memory error
2020-10-28 12:03:06 +08:00
Chris Lu
53c3aad875
volume: add a note file to avoid incomplete volume files
...
fix https://github.com/chrislusf/seaweedfs/issues/1567
2020-10-27 15:56:49 -07:00
Chris Lu
6da87720eb
shorter tag prefix
2020-10-27 15:33:26 -07:00
Chris Lu
24bf142596
copy large file first
2020-10-27 15:31:15 -07:00
Chris Lu
432c6f8db3
Merge pull request #1566 from taozix/master
...
return x-amz-tag-count header when GET object.
2020-10-27 13:16:58 -07:00
Chris Lu
06c15ab35c
volume: add special handling for .dat larger than 32GB
2020-10-27 13:11:56 -07:00
ruitao.liu
72f6b77dea
fix tag prefix.
2020-10-27 18:01:37 +08:00
ruitao.liu
8766ca1b95
rename s3 tag prefix.
2020-10-27 17:33:24 +08:00
ruitao.liu
19026ae55d
return x-amz-tag-count header when GET object.
2020-10-27 16:49:31 +08:00
Chris Lu
44921220b0
2.07
2020-10-25 22:03:46 -07:00
Chris Lu
e71463a9eb
mount: invalide file cache when metadata is changed
2020-10-25 19:24:15 -07:00
Chris Lu
e219c57849
passing full path when assign volume locations
2020-10-25 15:46:29 -07:00
Chris Lu
f375b93aef
renaming
2020-10-25 15:32:43 -07:00
Chris Lu
63b0fb54f7
minor
2020-10-25 14:15:53 -07:00
Chris Lu
27ba0a2be3
2.06
2020-10-25 12:36:47 -07:00
Chris Lu
147d4d23b6
filer.copy: automatically use the right bucket
2020-10-25 11:21:33 -07:00
Chris Lu
91fd311f7a
fix logic to read entry or not
2020-10-25 10:33:51 -07:00
Chris Lu
10a4a628e9
refresh cached file entry from sync metadata updates
2020-10-25 09:42:50 -07:00
Konstantin Lebedev
2fb1fce8a0
avoid old values when missing read only flags
2020-10-25 16:21:40 +05:00
Chris Lu
da36abf033
go fmt
2020-10-24 20:12:04 -07:00
Chris Lu
16fae84414
fix format error
2020-10-24 20:11:31 -07:00
Chris Lu
b7b1eb51b1
default to 2 for better performance in common computer hardware
2020-10-24 20:08:26 -07:00
Chris Lu
79690b9cd8
revert changes causing wrong data
2020-10-24 20:07:19 -07:00
Chris Lu
9104cfa744
reduce locks
2020-10-24 19:40:35 -07:00
Chris Lu
e9d40b80b2
less verbose logs
2020-10-24 09:42:54 -07:00
Chris Lu
6c4fb243a7
remove limits to concurrent writers
2020-10-24 02:05:05 -07:00
Chris Lu
e0002f8dd7
check existing volumes for writable status
2020-10-24 01:34:31 -07:00
Chris Lu
19772d70d7
print for debugging
2020-10-24 00:12:02 -07:00
Chris Lu
5b92dfe302
adjust log level
2020-10-23 23:58:54 -07:00
Chris Lu
fb124f2d4a
conditionally display volume server public url link
2020-10-23 23:58:46 -07:00
Chris Lu
1b17f71939
adjust election timeout to 10 seconds
2020-10-23 23:06:44 -07:00
Chris Lu
ef9c32ea0d
adjust compression threshold
2020-10-23 23:05:55 -07:00
Chris Lu
0cea84e7ab
make it easy with setting up multiple masters in a clcuster
2020-10-23 18:18:46 -07:00
Chris Lu
fc689319ae
add volume public url
2020-10-23 18:03:23 -07:00
Chris Lu
71624858af
avoid reusing cached []byte
...
this logic is just to ensure 100% correct.
fuse message, data+message hearder > 512 will not be cached.
2020-10-23 11:31:57 -07:00
Chris Lu
29c9fa2ef2
byte buffer for uploading
2020-10-23 11:29:51 -07:00
Chris Lu
2579edbc60
fix bug found by tests
2020-10-23 10:05:40 -07:00
Chris Lu
e0c8507d93
memory pool to reduce memory allocation
2020-10-23 00:41:40 -07:00
Chris Lu
b81359823f
postgres: support empty user
2020-10-22 14:27:47 -07:00
Chris Lu
11716fbf6f
make reading error more obvious
2020-10-22 13:33:45 -07:00
Chris Lu
d75bc62196
to read files that are quickly changing
2020-10-22 12:59:15 -07:00
Chris Lu
575d7952a1
add available resource stats
...
fix https://github.com/chrislusf/seaweedfs/issues/1555
2020-10-22 09:13:47 -07:00
Chris Lu
5179e559f7
skip empty logs
2020-10-22 00:35:48 -07:00
Chris Lu
de044e466e
Merge pull request #1553 from kmlebedev/multi_domainname
...
Multi domainname
2020-10-21 23:56:03 -07:00
Chris Lu
e73d6c9526
able to print partial file id
...
address https://github.com/chrislusf/seaweedfs/issues/1552
2020-10-21 23:48:07 -07:00
Konstantin Lebedev
a9a7005687
set desc of option
2020-10-22 11:23:00 +05:00
Chris Lu
720b1d9b88
adding locking to avoid nil VolumeLocationList
...
fix panic: runtime error: invalid memory address or nil pointer dereference
Oct 22 00:53:44 bedb-master1 weed[8055]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x17658da]
Oct 22 00:53:44 bedb-master1 weed[8055]: goroutine 310 [running]:
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLocationList).Length(...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_location_list.go:35
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLayout).enoughCopies(...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_layout.go:376
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLayout).ensureCorrectWritables(0xc000111d50, 0xc000b55438)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_layout.go:202 +0x5a
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*Topology).SyncDataNodeRegistration(0xc00042ac60, 0xc001454d30, 0x1, 0x1, 0xc0005fc000, 0xc00135de40, 0x4, 0xc00135de50, 0x10, 0x10d, ...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/topology.go:224 +0x616
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/server.(*MasterServer).SendHeartbeat(0xc000162700, 0x23b97c0, 0xc000ae2c90, 0x0, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/server/master_grpc_server.go:106 +0x325
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/pb/master_pb._Seaweed_SendHeartbeat_Handler(0x1f8e7c0, 0xc000162700, 0x23b0a60, 0xc00024b440, 0x3172c38, 0xc000ab7100)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/pb/master_pb/master.pb.go:4250 +0xad
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100, 0xc0001fea80, 0x311fec0, 0x0, 0x0, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1329 +0xcd8
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).handleStream(0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1409 +0xc5c
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0001ce8b0, 0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:746 +0xa5
Oct 22 00:53:44 bedb-master1 weed[8055]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:744 +0xa5
Oct 22 00:53:44 bedb-master1 systemd[1]: weedmaster.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 22 00:53:44 bedb-master1 systemd[1]: weedmaster.service: Failed with result 'exit-code'.
2020-10-21 23:15:48 -07:00
Chris Lu
2bc05cfbc8
fix typo
2020-10-21 22:26:41 -07:00
Chris Lu
4ff2ceee33
UI fix on rendering EC volumes
...
addressing UI problem with https://github.com/chrislusf/seaweedfs/issues/1551
2020-10-21 22:05:58 -07:00
Chris Lu
f5ee19e0db
shortcut for appending operations
2020-10-21 20:44:01 -07:00
Chris Lu
b5917bf846
clean up
2020-10-21 19:31:26 -07:00
Chris Lu
9884dfc369
use bytebuffer for interval list
2020-10-21 19:29:51 -07:00
Chris Lu
bd103c143a
add lock for vidCache
2020-10-21 19:28:59 -07:00
Konstantin Lebedev
05b5f12f2e
multiplate DomainNames through comma
2020-10-21 20:48:51 +05:00
Chris Lu
81cf8d04df
minor
2020-10-21 02:42:23 -07:00
Chris Lu
d90e7ee3cc
adjust log level
2020-10-21 02:17:40 -07:00
Chris Lu
3bf0116de1
mount: less channel waiting
2020-10-21 02:16:21 -07:00
Chris Lu
c31b254248
mount: shortcut when there is only one chunk
2020-10-20 23:48:29 -07:00
Chris Lu
f64252023e
Revert "a better byte buffer pool"
...
This reverts commit 59c6422777
.
2020-10-20 23:21:54 -07:00
Chris Lu
ad652ffb97
avoid close closed channel
2020-10-20 22:54:34 -07:00
Chris Lu
20442dc1bc
avoid extra data copying
2020-10-20 22:54:21 -07:00
Chris Lu
59c6422777
a better byte buffer pool
2020-10-20 22:53:58 -07:00
Chris Lu
6c92fa349d
filer: support default rack
...
fix https://github.com/chrislusf/seaweedfs/issues/1546
2020-10-20 17:41:39 -07:00
Chris Lu
b3aa2fab9a
s3: report error to s3 when updating an object but it is already a directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1545
2020-10-20 10:25:16 -07:00
Chris Lu
410b818aa7
master: avoid timer leakage
2020-10-19 14:24:57 -07:00
Chris Lu
69f0da5d73
mount: pass along replication and collection parameters
2020-10-19 00:12:42 -07:00
Chris Lu
c0ab458671
report nil instead of EOF for empty files
...
related to https://github.com/chrislusf/seaweedfs/issues/1541
2020-10-17 11:03:46 -07:00
Chris Lu
898db14729
2.05
2020-10-17 02:42:33 -07:00
Chris Lu
09bab17aff
mount: avoid "send on closed channel"
2020-10-16 12:54:23 -07:00
Chris Lu
9d80a3428c
add debug_webdav
2020-10-16 12:53:56 -07:00
Chris Lu
37f165d743
webdav: return io.EOF when at end of a file
...
fix https://github.com/chrislusf/seaweedfs/issues/1344
2020-10-16 12:53:37 -07:00
Chris Lu
68d39c86f1
mysql, postgres, cassandra: change kv key to base64 encoding
...
The exisitng key-value operation for stores using mysql, postgres, and maybe cassandra are already broken.
The kv is used to store hardlink, filer store signature and replication progress.
So users using hardlink and also uses mysql, postgres, or cassandra will have broken hard links.
Users using filer.sync will need to re-sync the files.
2020-10-16 11:10:12 -07:00
Chris Lu
ee1fc6558a
refactor
2020-10-16 11:02:44 -07:00
Chris Lu
06e6341097
set initial write time to avoid race condition with delayed chunks
2020-10-15 11:08:45 -07:00
Chris Lu
d598c47136
log errors
2020-10-15 11:01:43 -07:00
Chris Lu
5039aff310
release the goroutine to read errors
2020-10-15 10:59:18 -07:00
Chris Lu
ace0ea3d28
s3: avoid duplicated bucket
2020-10-15 10:52:20 -07:00
Konstantin Lebedev
dc2e13092d
add number of read only volumes metric
2020-10-15 15:48:40 +05:00
Chris Lu
e714c28a02
mount: async write file chunk
2020-10-14 23:28:03 -07:00
Chris Lu
c95d3d9dff
adjust tests
2020-10-14 12:27:52 -07:00
Chris Lu
93bcf56514
file read report EOF
...
fix https://github.com/chrislusf/seaweedfs/issues/1344
2020-10-14 12:18:24 -07:00
Chris Lu
1069b325dd
shell: volumeServer.evacuate adds printout for ec volumes
2020-10-13 20:26:03 -07:00
Chris Lu
0542911e29
go fmt
2020-10-13 19:50:46 -07:00
Chris Lu
58fa506491
minor
2020-10-13 19:50:22 -07:00
Chris Lu
28d4e1a51b
mount: retry for directory listing with filer
...
related to https://github.com/chrislusf/seaweedfs/issues/1530
2020-10-13 19:49:52 -07:00
Chris Lu
c127da1219
filer: linearize timeout for large chunk of data
2020-10-13 14:04:46 -07:00
Chris Lu
aac4cb1f0c
adds errror on read and write
2020-10-13 13:53:34 -07:00
Chris Lu
9b4f7fed14
mount: report filer IO error
...
related to https://github.com/chrislusf/seaweedfs/issues/1530
2020-10-13 11:21:13 -07:00
Chris Lu
3f7d1d1bf1
Only wait on retryable requests
2020-10-13 00:29:46 -07:00
Chris Lu
b18f21cce1
mount: fix bound tree with filer.path
...
fix https://github.com/chrislusf/seaweedfs/issues/1528
2020-10-12 21:58:37 -07:00
Chris Lu
f022aff289
add back http.StatusPartialContent
...
revert e7c04af1d0
2020-10-12 12:26:25 -07:00
Chris Lu
bbd0afd37e
2.04
2020-10-11 21:25:30 -07:00
Chris Lu
7704469d29
mount: outsideContainerClusterMode changed to use volume server publicUrl
2020-10-11 20:42:15 -07:00
Chris Lu
723ae11db4
refactoring in order to adjust volume server url later
2020-10-11 20:15:10 -07:00
Chris Lu
d155f907c2
mount: configurable read wait time
2020-10-10 20:09:43 -07:00
Chris Lu
9b0e8ef026
filer: added QR code to transfer files with mobile devices
2020-10-10 18:00:21 -07:00
Chris Lu
8a52379ecb
add retry if volume can not be found
2020-10-10 16:02:39 -07:00
Chris Lu
70af0ec24c
filer: fix hanging on read
2020-10-10 16:02:10 -07:00
Chris Lu
cff8bb6554
return proper error
2020-10-10 15:43:22 -07:00
Chris Lu
6da8eef54f
fix logic error
2020-10-09 00:01:47 -07:00
Chris Lu
5d01dd28a6
add logging
2020-10-08 23:38:00 -07:00
Chris Lu
8d34eb0050
mount:exponentially backoff if read error for about 10 minutes
2020-10-08 23:31:26 -07:00
Chris Lu
6e1f936efd
refactoring
2020-10-08 23:19:42 -07:00
Chris Lu
b2ee5873fb
fix error not being returned
2020-10-08 23:19:20 -07:00
Chris Lu
b067853162
mount: ignore recursion error
...
fix https://github.com/chrislusf/seaweedfs/issues/1514
the host OS should manage recursive deletion.
2020-10-08 21:55:55 -07:00
Chris Lu
e1a8e3f900
fix logging
2020-10-08 18:40:38 -07:00
Chris Lu
8676db0f6a
adjust logging
2020-10-08 18:33:37 -07:00
Chris Lu
bd8b9b0c2e
some changes to prepare fix after failed reads
2020-10-08 18:33:06 -07:00
Chris Lu
78859e35fc
s3: adjust config examples
2020-10-08 10:12:09 -07:00
Chris Lu
9e7a2772b1
s3: only admin can list all buckets
2020-10-08 10:12:09 -07:00
Chris Lu
eed492b73b
randomize file locations
2020-10-07 23:58:32 -07:00
Chris Lu
ec08a1670b
volume: avoid hanging when stopping a volume server
2020-10-07 23:48:24 -07:00
Chris Lu
62d3d3aea0
go fmt
2020-10-07 23:30:54 -07:00
Chris Lu
e91b9c85a2
s3: Added support for "List" action in weed s3 -config=... in the config file.
...
fix https://github.com/chrislusf/seaweedfs/issues/1511
2020-10-07 23:22:35 -07:00
Chris Lu
a8624c2e4f
read from alternative replica
...
related to https://github.com/chrislusf/seaweedfs/issues/1512
2020-10-07 22:49:04 -07:00
Chris Lu
da4edf3651
master: check peers for existing leader before starting a leader election
...
fix https://github.com/chrislusf/seaweedfs/issues/1509
2020-10-07 01:25:39 -07:00
Chris Lu
c543762e23
volume: auto expands -max setting
...
I have tried to run weed volume but got errors...
cmd:
weed volume -max=0 -mserver="127.0.0.1:9333" -port=8080 -dir="/var/www/d1/sfs,/var/www/d2/sfs,/var/www/d3/sfs"
log:
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d1/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d2/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d3/sfs Permission: -rwxr-xr-x
F1006 15:09:26 17040 volume.go:142] 3 directories by -dir, but only 1 max is set by -max
What am I doing wrong? Thx.
2020-10-06 09:05:30 -07:00
Chris Lu
36492c47ec
adjust
2020-10-05 14:06:18 -07:00
Chris Lu
8f8e9ddbde
adjust API
2020-10-05 09:47:07 -07:00
Konstantin Lebedev
e4f2d9eb4a
We return etag using the same algorithm as aws s3
...
https://teppen.io/2018/06/23/aws_s3_etags/
2020-10-05 14:43:32 +05:00
Chris Lu
4fc673341f
2.03
2020-10-04 21:58:22 -07:00
Chris Lu
6b591b02af
adjust log level
2020-10-04 16:21:43 -07:00
Устюжанин Антон Александрович
702b1cb876
fix: remove deleted peers if resumeState = true
2020-10-04 21:56:17 +05:00
Chris Lu
d522df0ecd
adjust log level
2020-10-04 01:35:32 -07:00
Chris Lu
666859f7cd
mount: read prefetching
2020-10-04 01:31:04 -07:00
Chris Lu
bc47835997
mount: avoid duplicated reads
2020-10-03 20:16:42 -07:00
Chris Lu
8d65ad1444
Revert "mount: adds read prefetching"
...
This reverts commit 692f0614d1
.
2020-10-03 19:40:56 -07:00
Chris Lu
692f0614d1
mount: adds read prefetching
2020-10-03 16:50:09 -07:00
Chris Lu
707936f482
re-enable caching larger than 16MB
...
revert 62ce85610e
2020-10-03 14:12:38 -07:00
Chris Lu
2c20ef72ae
fix starting problem
2020-10-03 13:56:38 -07:00
Chris Lu
d77e3c96e9
FUSE: use the mount directory permissions
2020-10-03 13:37:35 -07:00
Chris Lu
95ffed87c0
Merge pull request #1505 from ustuzhanin/remove_raft_state
...
Resume raft state
2020-10-03 10:38:47 -07:00
Устюжанин Антон Александрович
6fbfe73683
fix: resumeState
2020-10-03 22:16:47 +05:00
Устюжанин Антон Александрович
dc31b19469
fix: restore raft state
2020-10-03 14:03:41 +05:00
Chris Lu
f781cce500
s3: support object tagging
...
* GetObjectTagging
* PutObjectTagging
* DeleteObjectTagging
2020-10-02 22:21:51 -07:00
Устюжанин Антон Александрович
8c82fb7e5f
fix: restore raft state
2020-10-02 23:01:20 +05:00
Chris Lu
9ab98fa912
s3 metrics adjust the label
2020-10-01 07:17:47 -07:00
Chris Lu
f1e879fe7a
2.02
2020-10-01 07:15:49 -07:00
Chris Lu
a1c01d716b
volume: avoid deadlock when deleting volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/1501
2020-10-01 07:10:03 -07:00
Konstantin Lebedev
86329bbf2b
label name is statusCode
2020-10-01 01:22:38 +05:00
Konstantin Lebedev
68463e92c1
add status code in S3RequestCounter
2020-10-01 00:59:39 +05:00
Chris Lu
79ab10e300
adjust help message
2020-09-30 09:32:00 -07:00
Chris Lu
55cb68c09c
filer: replication follows master config if not specified
2020-09-30 09:15:55 -07:00
Chris Lu
b9887504e8
fix test
2020-09-27 23:19:50 -07:00
Chris Lu
f46eae284e
adjust for test
2020-09-27 23:08:11 -07:00
Chris Lu
75aca5e13d
2.01
2020-09-27 23:02:41 -07:00
Chris Lu
e6552b5e1e
filer: able to start s3 together
2020-09-27 23:00:43 -07:00
Chris Lu
474e2b6ac3
add a hard link marker to 16byte + maker, for future extensions
2020-09-27 22:38:30 -07:00
Chris Lu
c49e2bb9a3
adjust
2020-09-27 12:07:45 -07:00
Chris Lu
62ce85610e
skip caching too large chunks
2020-09-27 11:58:48 -07:00
Chris Lu
9ad2dcca2b
more tests
2020-09-27 11:42:51 -07:00
Chris Lu
e43d86c796
fix pre allocated volume size
2020-09-27 10:58:19 -07:00
Chris Lu
31fc7bb2e1
refactor
...
adjust for faster test
2020-09-27 10:41:29 -07:00
Chris Lu
a37535cd9f
avoid non utf-8 in filename
...
fix https://github.com/chrislusf/seaweedfs/issues/1493
2020-09-26 00:13:39 -07:00
Chris Lu
41be7a4c8f
filer: upload also set file size
...
ensure works same as fuse mount
2020-09-25 12:02:06 -07:00
Chris Lu
35f6518c36
weed upload: usePublicUrl was not being used on big files
...
fix https://github.com/chrislusf/seaweedfs/issues/1492
2020-09-25 02:41:38 -07:00
Chris Lu
22049dc2e6
Merge pull request #1491 from hilimd/master
...
s3: fix get object metadata problem
2020-09-25 02:35:00 -07:00
limd
402aef8f30
s3:
...
1.fix spark reading S3 directory wildcard problem
2.fix the problem of the spark history service writing S3 directory
reference git revsion number: b41b7ea4d0
2020-09-25 14:37:02 +08:00
Chris Lu
8516517c48
filer store: Cassandra supports username/password
2020-09-24 21:31:06 -07:00
limd
ed7816681a
Add cassandra authenticator mode
2020-09-25 11:22:47 +08:00
limd
370a98cf6b
code style
2020-09-25 11:19:42 +08:00
limd
aee3fd08b9
code style
2020-09-25 11:04:32 +08:00
Chris Lu
043b063136
Merge pull request #1482 from hilimd/master
...
Fix: s3 delete object
2020-09-24 18:21:34 -07:00
limd
48c578410f
mount: rollback default value
2020-09-25 09:18:52 +08:00
Chris Lu
dbf5327b98
s3: handle response-content-encoding
...
fix https://github.com/chrislusf/seaweedfs/issues/1487
2020-09-24 18:09:52 -07:00
Chris Lu
7726965a47
minor
2020-09-24 11:20:12 -07:00
Chris Lu
0790c6d605
fix empty hard link id
2020-09-24 11:16:43 -07:00
Chris Lu
1012df7bb5
switch hardlink id from int64 to bytes
2020-09-24 11:11:42 -07:00
Chris Lu
4856bce0ee
adjust for metrics port
2020-09-24 10:21:23 -07:00
Chris Lu
d2d3aec3e1
consolidate to one metricsPort in "weed server" mode
2020-09-24 09:55:02 -07:00
Chris Lu
2e9099369e
Merge pull request #1489 from kmlebedev/promhttp
...
Promhttp
2020-09-24 09:50:48 -07:00
Chris Lu
2e7c361a0d
hardlink deletion factors in hardlink counter
2020-09-24 09:43:52 -07:00
Chris Lu
1295347958
adjust hardlink update
...
simplify logic, pass entity content directly to hard link. The "weed mount" handles the logic to calculate hard link counter.
2020-09-24 09:43:00 -07:00
Konstantin Lebedev
98e9de6e11
fix style
2020-09-24 17:48:39 +05:00
Konstantin Lebedev
324e44d4b3
add start metrics server
2020-09-24 17:45:39 +05:00
limd
cbca14edc5
mount: fix k8s pvc and os mount directory permission bug
2020-09-24 18:07:16 +08:00
Chris Lu
5e239afdfc
hardlink works now
2020-09-24 03:06:48 -07:00
limd
59e91e9c7e
mount: fix k8s pvc mount directory permission
2020-09-24 10:25:45 +08:00
Chris Lu
c7d7b1a0f6
Merge pull request #1485 from LIBA-S/fix_oversized
...
Correct the oversized state of volume after compaction
2020-09-23 19:24:30 -07:00
Chris Lu
b80a2b3cc9
Merge pull request #1486 from LIBA-S/fix_race_condition
...
Fix a race condition when handle VolumeLocationList
2020-09-23 19:24:16 -07:00
LIBA-S
eecd6b5d35
Fix a race condition when handle VolumeLocationList
2020-09-23 20:56:51 +08:00
LIBA-S
0157798ebf
Correct the oversized state of volume after compaction
2020-09-23 20:27:42 +08:00
limd
9b21ec27e4
mount: auto created dir set corrct umask fix bug
2020-09-23 18:35:37 +08:00
hilimd
0db149fb5f
Merge pull request #18 from chrislusf/master
...
sync
2020-09-23 17:32:05 +08:00
Chris Lu
f7a0ccb595
mount: auto created dir set corrct umask
2020-09-23 02:31:19 -07:00
Chris Lu
f0e325b6fc
mount: auto created directory follow umask
2020-09-23 02:27:57 -07:00
limd
d506080c36
rollback
2020-09-23 14:29:53 +08:00
limd
8f9f29b773
fixed the problem of empty directory when S3 deleted the directory
2020-09-23 13:33:13 +08:00
limd
1892677b22
fixed the problem of empty directory when S3 deleted the directory
2020-09-23 13:15:06 +08:00
hilimd
85570f27a0
Merge pull request #17 from chrislusf/master
...
sync
2020-09-23 11:39:57 +08:00
limd
a99f63cb4d
1.add S3 copy directory function
...
2.fixed the problem of empty directory when S3 deleted the directory
2020-09-23 11:26:01 +08:00
Chris Lu
b61d33f251
expose only store wrapper to meta changes
2020-09-22 16:27:36 -07:00
Chris Lu
0adbb56cc1
rename
2020-09-22 16:24:13 -07:00
Chris Lu
3e52329cee
Revert "Merge pull request #1479 from LIBA-S/fix_oversized"
...
This reverts commit bd11f0b3e4
, reversing
changes made to ec5b9f1e91
.
2020-09-22 15:05:37 -07:00
Chris Lu
4a1fe4b8e2
add error logs
2020-09-22 09:16:10 -07:00
LIBA-S
8573ac82c6
Fix: remove the oversized state after compaction
2020-09-22 21:48:39 +08:00
James Hartig
658fc2e5b6
Allow option to enable volume pprof on server
2020-09-21 22:43:10 -04:00
James Hartig
91e4eca1e9
Fix deadlock with KeepConnected and SendHeartbeat
...
There's the potential where we're writing to a clientConn and it goes away
and we're stuck keeping a read lock on clientChansLock. This causes
KeepConnected to not be able to remove the client since it requires a write
lock on clientChansLock. This ends up backing up SendHeartbeat because it
can't get a read lock.
2020-09-21 22:41:38 -04:00
Chris Lu
9cdbfc1a49
refactor
2020-09-21 17:34:38 -07:00
Chris Lu
63373a9f9f
filer: add file extended properties
2020-09-21 11:08:34 -07:00
Chris Lu
b9c1f3e9de
s3: fixes for list multipart upload
2020-09-21 10:51:24 -07:00
Chris Lu
289e62a305
master: better locking of in memory volume data
...
related to https://github.com/chrislusf/seaweedfs/issues/1436#issuecomment-695880135
2020-09-20 23:07:55 -07:00
Chris Lu
9a3b564508
adjust printout
2020-09-20 16:01:56 -07:00
Chris Lu
62563a895a
refactoring
2020-09-20 16:00:01 -07:00
Chris Lu
c2faab23b6
refactor
2020-09-20 15:40:49 -07:00
Chris Lu
d013d09a9b
adjust logging
2020-09-20 15:38:59 -07:00
Chris Lu
f498c71199
shell: move volume operations to use flag parsing arguments
2020-09-20 09:27:34 -07:00
Chris Lu
fcbc520373
fix print out
2020-09-20 08:47:30 -07:00
Chris Lu
56094541c8
shell: adjust command option for collection.delete
2020-09-20 08:46:16 -07:00
Chris Lu
9cac9c4001
2.00
2020-09-20 00:47:04 -07:00
Chris Lu
dc4d2145ff
open up
2020-09-19 20:23:12 -07:00
Chris Lu
29abe980df
s3: add support for PostPolicy
...
fix https://github.com/chrislusf/seaweedfs/issues/1426
2020-09-19 20:14:19 -07:00
Chris Lu
41d508edfd
go fmt
2020-09-19 14:10:26 -07:00
Chris Lu
5b40a2690a
refactoring
2020-09-19 14:09:58 -07:00
Chris Lu
2c21eb1971
volume: get metrics configuration from master
...
fix https://github.com/chrislusf/seaweedfs/issues/1354
2020-09-19 00:03:00 -07:00
Chris Lu
2cbd1cf121
fix compilation
2020-09-18 00:15:54 -07:00
Chris Lu
23e9ede068
s3: collect metrics
2020-09-18 00:09:04 -07:00
Chris Lu
852e5f7cbc
filer: fix mongodb insert
...
fix https://github.com/chrislusf/seaweedfs/issues/1471
2020-09-17 21:50:52 -07:00
Chris Lu
6544e60bea
s3 add metrics
...
empty for now
2020-09-17 06:56:15 -07:00
Chris Lu
cb427d48fa
filer report metrics configuration
2020-09-17 06:46:51 -07:00
Chris Lu
e4e0234998
refactoring
2020-09-17 06:43:54 -07:00
Chris Lu
feca07bf96
Filer: ip bind
...
fix https://github.com/chrislusf/seaweedfs/issues/1470
2020-09-16 10:26:08 -07:00
Chris Lu
e861a6a3ab
simplify metrics configuration logic
2020-09-16 01:39:30 -07:00
Chris Lu
be54eeb364
passing value by checkWithMaster() instead
2020-09-16 01:33:45 -07:00
Chris Lu
4d21de63ee
go fmt
2020-09-16 01:27:24 -07:00
Chris Lu
913a16268d
volume: load configuration from master at the start
...
fix https://github.com/chrislusf/seaweedfs/issues/1469
2020-09-16 01:27:05 -07:00
Chris Lu
c9202c4b3d
add storage backend to GetMasterConfigurationResponse
2020-09-16 01:12:22 -07:00
Chris Lu
8a0710cb73
handle more than 2GB files
...
related to https://github.com/chrislusf/seaweedfs/issues/1468
2020-09-16 00:41:50 -07:00
Chris Lu
854007bc98
minor
2020-09-16 00:37:57 -07:00
Chris Lu
10f9081526
filer: adjust meta data events to received timestamp
...
if a client is already connected and start from t0. A message recieved at t+1 but with timestamp t-1 may not be processed by the client.
This commit changes to the event received time, so the replication can be ordered.
2020-09-15 01:18:33 -07:00
Chris Lu
684a875876
update help message
2020-09-15 00:40:38 -07:00
Chris Lu
2d3a904a82
shell: volumeServer.evacuate adds option to skip non moveable volumes
2020-09-15 00:33:49 -07:00
Chris Lu
a595916342
shell: add volumeServer.evacuate command
2020-09-14 23:47:11 -07:00
Chris Lu
103fafe00b
weed export: print out [start,stop) content range
2020-09-14 22:57:23 -07:00
Chris Lu
ada996fe55
filer: support createing empty folder
...
fix https://github.com/chrislusf/seaweedfs/issues/1161
2020-09-14 13:07:46 -07:00
Chris Lu
200fe5c83e
go fmt
2020-09-13 21:26:30 -07:00
Chris Lu
5d6753fb98
shell: add volumeServer.leave command
2020-09-13 21:25:51 -07:00
Chris Lu
1af95c5b76
refactoring
2020-09-13 12:41:26 -07:00
Chris Lu
f8fea19669
1.99
2020-09-12 13:46:33 -07:00
Chris Lu
b0c7de186d
filer: fix postgres prefixed directory listing problem
...
fix https://github.com/chrislusf/seaweedfs/issues/1465
2020-09-12 13:37:03 -07:00