Chris Lu
66cd219b15
Merge branch 'master' into support_ssd_volume
2020-12-13 04:08:36 -08:00
Chris Lu
813453f3e1
2.15
2020-12-13 03:45:35 -08:00
Chris Lu
17372ac3da
balance ssd volumes
2020-12-13 03:40:33 -08:00
Chris Lu
d156c74ec0
volume server set volume type and heartbeat to the master
2020-12-13 03:11:24 -08:00
Chris Lu
e9cd798bd3
adding volume type
2020-12-13 00:58:58 -08:00
Chris Lu
16cd6fb278
fix variable name
2020-12-12 21:19:40 -08:00
Chris Lu
5c465293e9
correctly determine whether a folder is empty
...
avoid edge cases that deleting the folder if previous 32 directories are all empty
early terminate if one file is found
2020-12-12 16:19:29 -08:00
Chris Lu
14910d035c
the pagination size was too big for recursive deletion
2020-12-12 13:26:10 -08:00
Chris Lu
f930c713fc
more efficient recursion
2020-12-12 13:25:19 -08:00
Chris Lu
316d1b4e69
refactor APIs
2020-12-12 12:42:53 -08:00
Chris Lu
03637d6f57
s3: move "delete-directory-if-empty" to read time
...
move "delete-directory-if-empty" to read time instead of entry deletion time
the listing speed for a s3 bucket folder will slow down if it has many sub folders
related to 0d345ac97d
fix https://github.com/chrislusf/seaweedfs/issues/1647
fix https://github.com/chrislusf/seaweedfs/issues/1670
2020-12-12 03:38:34 -08:00
Chris Lu
37075a414d
adjust logs
2020-12-12 03:33:57 -08:00
Chris Lu
ae655033ac
adjust logging
2020-12-11 16:57:53 -08:00
Chris Lu
d986c7196d
use append time instead of filer's own modification time
...
fix https://github.com/chrislusf/seaweedfs/issues/1669
2020-12-11 16:55:18 -08:00
Chris Lu
3fedfec1e7
check cross device rename error
2020-12-10 23:50:32 -08:00
Chris Lu
c2f18a10cb
minor
2020-12-10 22:23:22 -08:00
Chris Lu
4b0c2a846b
add logs
2020-12-10 19:55:28 -08:00
Chris Lu
83078ac6ce
filer: change to /etc/seaweedfs folder on filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1666
2020-12-10 11:11:02 -08:00
Chris Lu
26731694f8
s3: use static configuration by default
...
So that users can still use the previous configuration files.
If leave it empty, s3 will try to use the version from filer
2020-12-10 00:59:04 -08:00
Chris Lu
765b3ef1be
save /etc/iam/identity.json inside filer store
2020-12-10 00:15:22 -08:00
Chris Lu
97c942b5ee
paginate through large directories
2020-12-09 23:23:38 -08:00
Chris Lu
02a160c3fd
s3: recursively iterate all sub folders
...
fix https://github.com/chrislusf/seaweedfs/issues/1656
2020-12-09 16:47:34 -08:00
Chris Lu
a9c619c270
added error handling for saving
2020-12-09 01:34:24 -08:00
Chris Lu
4211601eab
set file handle reader to nil
2020-12-08 22:48:18 -08:00
Chris Lu
b52ae9cef8
add s3 debug
2020-12-08 22:27:04 -08:00
Chris Lu
8e78187a97
add back last read chunk cache to reader and properly close the reader
2020-12-08 22:26:46 -08:00
Chris Lu
900d22c6ec
mount: avoid memory leaking read buffer
...
fix https://github.com/chrislusf/seaweedfs/issues/1654
the reader goes together with the file handle, which may stay for a long time.
2020-12-08 02:38:53 -08:00
Chris Lu
06bb7bf6c0
adding /usr/local/etc/seaweedfs for bsd style config search path
...
related to https://reviews.freebsd.org/D27391
2020-12-07 16:46:48 -08:00
Chris Lu
fce8803087
break import cycle
2020-12-07 00:29:17 -08:00
Chris Lu
606051cc79
filer: cache "/etc" filder small files to filer store
2020-12-07 00:11:35 -08:00
Chris Lu
ac22f1dd26
filer subscribe: handle rename subscription
2020-12-07 00:10:49 -08:00
Chris Lu
eed87791b7
s3: subscribe to s3.configure changes
2020-12-07 00:10:29 -08:00
Chris Lu
01e2da5782
refactoring
2020-12-06 23:16:20 -08:00
Chris Lu
77286f8bea
fix test
2020-12-06 21:56:13 -08:00
Chris Lu
57578a6cd1
change parameter help message
2020-12-06 21:54:55 -08:00
Chris Lu
78f3ab439c
refactoring
2020-12-06 21:54:34 -08:00
Chris Lu
9acda432fe
fix import cycle
2020-12-06 20:12:52 -08:00
Chris Lu
ae5eb85a06
refactoring
2020-12-06 20:05:06 -08:00
Chris Lu
5931a2f53f
filer conf: support json conf only
2020-12-06 19:47:06 -08:00
Chris Lu
4c72482536
shell: fix fs.configure
2020-12-06 19:43:31 -08:00
Chris Lu
46b91228af
2.14
2020-12-06 18:56:58 -08:00
Chris Lu
263eb29e9f
filer: add option to cache small files to filer store
2020-12-04 22:39:43 -08:00
Chris Lu
d171d9f988
volume: ensure the volume dat and idx files are always closed
...
fix https://github.com/chrislusf/seaweedfs/issues/1646
2020-12-04 21:50:26 -08:00
Konstantin Lebedev
eadbba5c95
security master use access.ui
2020-12-04 12:57:59 +05:00
Chris Lu
cc839f935d
better unit detection
2020-12-03 19:23:59 -08:00
Chris Lu
b219ccfe68
Update volume_ttl_test.go
2020-12-03 12:55:44 -08:00
Chris Lu
a83b8e7b01
ttl fix: 3y was converted into 96m
2020-12-03 10:53:29 -08:00
Chris Lu
5fb60b7135
Merge pull request #1596 from kmlebedev/store_s3cred
...
S3 credentials store in filer
2020-12-03 00:40:20 -08:00
Konstantin Lebedev
14699dfcef
use content field of entry
2020-12-02 17:19:05 +05:00
Konstantin Lebedev
a3d4b50a49
use entry content filed
2020-12-02 16:12:13 +05:00
Chris Lu
003b6245e7
fix nil
2020-12-02 00:09:19 -08:00
Chris Lu
f3bb645018
file open error
2020-12-01 23:37:49 -08:00
Chris Lu
dc0bc48257
return file open error
2020-12-01 23:36:49 -08:00
Chris Lu
45dba088cd
fix tests
2020-12-01 19:37:21 -08:00
Chris Lu
04062c56c7
webdav: improve webdav upload speed
2020-12-01 15:32:27 -08:00
Konstantin Lebedev
03620776ec
Merge branch 'upstream_master' into store_s3cred
2020-12-01 16:03:34 +05:00
Chris Lu
005a6123e9
Merge pull request #1643 from hilimd/master
...
fix tls grpc ca path
2020-11-30 17:16:46 -08:00
Chris Lu
f4abd01adf
filer: cache small file to filer store
2020-11-30 04:34:04 -08:00
Chris Lu
a9c6be5fc3
set creation time
2020-11-30 03:11:52 -08:00
Chris Lu
141ce67c09
close http request body
2020-11-30 02:45:00 -08:00
Chris Lu
0e99531dbf
2.13
2020-11-29 17:00:03 -08:00
Chris Lu
965413c21b
shell: add volume.vacuum command
2020-11-28 23:18:02 -08:00
Chris Lu
96c48bc8a8
fix test
2020-11-28 03:03:43 -08:00
Chris Lu
c57a7839ec
adjust comments
2020-11-28 03:01:03 -08:00
Chris Lu
8438fb0cc3
fix ec shard spreading error
2020-11-28 02:21:16 -08:00
Chris Lu
536202e4e0
minor
2020-11-28 02:16:06 -08:00
Chris Lu
99c4e50d3d
minor
2020-11-28 00:14:11 -08:00
Chris Lu
2c913dde04
volume: detect and drop volumes with disk IO error
...
from Jethro in slack:
is it possible to make the assign request a bit smarter? Currently I’m in the state that a disk failed but all assign request are being send to this volume. It would be cool if the master sees this and stopped using this volume.
e=HTTP(http://x:8089/913,045a782b63176edf ) not 200 but 500 Internal Server Error
Body={"size":740167,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"ee4381e202212ff3aee647704c036689"}
e=HTTP(http://x:8089/913,045a782c90240077 ) not 200 but 500 Internal Server Error
Body={"size":792779,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"c43463ccc11eb6eb2fc306f407a6a953"}
e=HTTP(http://x:8089/913,045a782e6b7901ea ) not 200 but 500 Internal Server Error
Body={"size":3962392,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"04c91198e9b276c81f11dbf189af5d28"}
2020-11-28 00:09:29 -08:00
Chris Lu
9ac4935f22
read from volume index file directly instead of open a separate file
...
fix https://github.com/chrislusf/seaweedfs/issues/1640
read from volume index file directly instead of open a separate file,
to ensure reading latest index entries.
2020-11-27 16:18:48 -08:00
Chris Lu
85554bea38
filer: readonly handle static resources
...
fix https://github.com/chrislusf/seaweedfs/issues/1641
2020-11-27 13:42:14 -08:00
Chris Lu
6d30b21b10
volume: add "-dir.idx" option for separate index storage
...
fix https://github.com/chrislusf/seaweedfs/issues/1265
2020-11-27 03:17:10 -08:00
Chris Lu
3c229eb677
refactor
2020-11-26 17:21:55 -08:00
Chris Lu
c0fb4a3766
add comments
2020-11-26 15:22:42 -08:00
Chris Lu
6783ae5f8f
rename
2020-11-26 15:19:43 -08:00
Chris Lu
983fd90010
marked as WIP
2020-11-26 15:08:52 -08:00
Chris Lu
97f706f35b
just refactoring
2020-11-26 15:08:05 -08:00
Chris Lu
b11449f955
pass in dir.idx parameter to DiskLocation
2020-11-26 14:59:03 -08:00
Chris Lu
0d345ac97d
s3: remove empty parent folder on delete
...
fix https://github.com/chrislusf/seaweedfs/issues/1637
2020-11-26 12:21:58 -08:00
Chris Lu
cc2bd97ad9
refactor
2020-11-26 11:25:56 -08:00
Chris Lu
0da7ecfd29
go fmt
2020-11-26 11:22:30 -08:00
Chris Lu
effa00ed08
refactor
2020-11-26 11:21:58 -08:00
Chris Lu
513bcd6e0d
filer: avoid duplicated FindEntry for deletion
2020-11-26 11:14:56 -08:00
Chris Lu
1ae108efca
add debug message
2020-11-26 10:41:10 -08:00
Chris Lu
a102157020
do not retry if failed with a non transport related error
2020-11-26 01:29:30 -08:00
Konstantin Lebedev
4e55baf5b1
s3 config changes
2020-11-26 03:50:53 +05:00
Konstantin Lebedev
52c8f2fc9a
s3iam test
2020-11-26 02:26:45 +05:00
Konstantin Lebedev
a26f1b2040
new pkg s3iam
2020-11-26 01:30:11 +05:00
Konstantin Lebedev
6206737df2
s3 configure
2020-11-25 21:02:31 +05:00
Chris Lu
eab53ea80d
filer leveldb store: a bit more efficient directory listing with prefix
2020-11-22 21:10:41 -08:00
Chris Lu
d3be2ceedd
2.12
2020-11-22 17:17:33 -08:00
Chris Lu
c7ebadc25d
avoid possible concurrent access inside ensureCorrectWritables()
2020-11-22 17:15:59 -08:00
hilimd
9aa990f80f
Merge pull request #39 from chrislusf/master
...
sync
2020-11-22 20:29:17 +08:00
limd
e8296104fc
fix tls grpc ca path
2020-11-22 20:27:15 +08:00
Chris Lu
92f906b6fc
remove zstd
...
fix https://github.com/chrislusf/seaweedfs/issues/1629
2020-11-21 13:06:45 -08:00
Aleksandr Demshin
176950ea78
fix typo
2020-11-20 19:50:46 +05:00
Chris Lu
7385103057
fix volume placement validating
...
fix https://github.com/chrislusf/seaweedfs/issues/1626
2020-11-20 01:12:25 -08:00
Konstantin Lebedev
27e73de797
Merge branch 'upstream_master' into store_s3cred
...
# Conflicts:
# weed/s3api/filer_util.go
2020-11-19 18:16:44 +05:00
Chris Lu
290b5e2cd0
directly delete file chunks
...
keeping current async deletions for now
2020-11-17 17:20:21 -08:00
Chris Lu
781585b195
remove unnecessary checking
2020-11-17 17:00:06 -08:00
Chris Lu
8cb8cd4cc5
add locks
2020-11-17 16:59:48 -08:00
Chris Lu
7c9e592c2d
s3: avoid nil resp when having error
...
fix https://github.com/chrislusf/seaweedfs/issues/1622
2020-11-17 11:23:17 -08:00
Chris Lu
dc304342b2
fs.configure: configurable volume growth
2020-11-17 01:00:02 -08:00
Chris Lu
b04375a3d1
use alternative assign request
2020-11-17 00:36:21 -08:00
Chris Lu
9add554feb
fail fast if user configures collection for a bucket
2020-11-16 19:57:08 -08:00
Chris Lu
6204dc6e85
add comment
2020-11-16 16:57:31 -08:00
Chris Lu
ef3e068c51
deprecating "filer.options.buckets_fsync" in filer.toml
2020-11-16 16:56:49 -08:00
Chris Lu
5f19e81dab
filer confi: support hierachical configuration
2020-11-16 16:50:12 -08:00
Chris Lu
ed1ce3f299
adjust help message
2020-11-16 09:54:26 -08:00
Chris Lu
c0d279c54e
filere.conf: prefer to use json format
2020-11-15 21:48:17 -08:00
Chris Lu
83527a8f55
adds more help message
2020-11-15 21:31:14 -08:00
Chris Lu
af658ea970
fix typo
2020-11-15 21:16:28 -08:00
Chris Lu
98827d68d9
2.11
2020-11-15 20:25:56 -08:00
Chris Lu
ee2fa14dbe
filer conf: delete location specific configuration
2020-11-15 20:15:47 -08:00
Chris Lu
2bd6fd3bbe
remove unused function
2020-11-15 20:15:07 -08:00
Chris Lu
71056dae07
fs.configure: read and local add filer configuration
2020-11-15 18:09:35 -08:00
Chris Lu
0ea5c087ce
go fmt
2020-11-15 16:59:28 -08:00
Chris Lu
95c0de285d
refactoring
2020-11-15 16:58:48 -08:00
Chris Lu
500bcab953
refactoring
2020-11-15 14:41:56 -08:00
Chris Lu
590f02179d
filer: load filer conf when starting
2020-11-15 14:06:03 -08:00
Chris Lu
0a406f652e
load filer conf and match by prefix
2020-11-15 00:26:05 -08:00
Chris Lu
68043cfcac
add reference implementation to detect create/update/delete/rename events
2020-11-14 21:21:58 -08:00
Chris Lu
0fc9ffc603
able to read filer.conf if updated
2020-11-14 21:21:20 -08:00
Chris Lu
442e092995
filer watch "/etc" folder for configuration changes
2020-11-14 14:26:08 -08:00
ruitao.liu
a9990a1dc6
adjust check bucket if exist or has access.
2020-11-13 17:13:20 +08:00
Chris Lu
e6333da65a
enable admin to access all buckets
2020-11-12 13:57:54 -08:00
Chris Lu
7094492428
fix ErrNotFound
2020-11-12 13:30:46 -08:00
Chris Lu
c6a0704019
rename function
2020-11-12 13:30:08 -08:00
Chris Lu
ed3b0f81c6
Merge pull request #1613 from taozix/master
...
check permission for bucket delete/head.
2020-11-12 13:25:28 -08:00
ruitao.liu
c4f0fd6e1b
skip if entry.Extended map is nil.
2020-11-12 17:59:31 +08:00
ruitao.liu
ab966410d2
return NoSuchBucket instead of InternalError delete non-existed bucket.
2020-11-12 16:44:16 +08:00
Chris Lu
a4dfa00030
adjust filer conf
2020-11-12 00:39:59 -08:00
Chris Lu
2251a8e472
minor
2020-11-12 00:38:58 -08:00
Chris Lu
5a16f17e47
remove unused message type
2020-11-12 00:38:23 -08:00
ruitao.liu
e06676f007
check permission for bucket delete/head.
2020-11-12 16:15:59 +08:00
Chris Lu
c32b6f4d9d
prepare for path based configuration
2020-11-11 23:46:45 -08:00
Chris Lu
0fbc27b640
Merge pull request #1612 from taozix/master
...
add owner attr for bucket.
2020-11-11 22:19:47 -08:00
ruitao.liu
d7cc0498e0
check if bucket already exists.
2020-11-12 14:11:03 +08:00
Chris Lu
827930e249
Merge pull request #1610 from kmlebedev/prefer_read_in_this_dc
...
prefer to read from volumes in this data center
2020-11-11 22:09:41 -08:00
Konstantin Lebedev
0880aff224
update dc flag desc
2020-11-12 10:07:52 +05:00
ruitao.liu
dbba8cb57e
fix if nil map in entry.
2020-11-12 11:50:19 +08:00
Konstantin Lebedev
aa30604cec
fix tests
2020-11-12 08:49:38 +05:00
Konstantin Lebedev
1eec5c8d5d
gen pb
2020-11-12 04:10:06 +05:00
Konstantin Lebedev
fc7baef5bb
fiil serverUrls sorted by data center
2020-11-12 02:13:33 +05:00
Chris Lu
745ee8d8f3
avoid unnecessary error message
...
fix https://github.com/chrislusf/seaweedfs/issues/1611
2020-11-11 13:00:05 -08:00
Chris Lu
a29abd8600
add a {X-,}Idempotency-Key header for http POST
...
fix https://github.com/chrislusf/seaweedfs/issues/1592
according to https://github.com/golang/go/issues/19943
2020-11-11 12:56:53 -08:00
Chris Lu
8708e7a64d
print out data size
2020-11-11 12:51:44 -08:00
Chris Lu
c6bd244ebd
add TODO
2020-11-11 12:51:27 -08:00
Konstantin Lebedev
b73ef6aa95
Merge remote-tracking branch 'upstream/master' into prefer_read_in_this_dc
2020-11-11 23:29:57 +05:00
Konstantin Lebedev
dc26012a3b
initial
2020-11-11 15:03:47 +05:00
Chris Lu
86cdb2a3e5
adjust logs
2020-11-11 02:01:24 -08:00
Chris Lu
73f934d5de
s3: do not close reader too early
...
fix https://github.com/chrislusf/seaweedfs/issues/1609
2020-11-11 01:49:39 -08:00
ruitao.liu
5b636b3242
Add bucket owner attr.
2020-11-11 16:20:59 +08:00
Chris Lu
15956d8a8e
2.10
2020-11-10 23:17:13 -08:00
Chris Lu
de3bdd0651
delete old volume replica
...
related to https://github.com/chrislusf/seaweedfs/issues/1607
old is:
* older compaction revision
* older modified time
* smaller volume size
2020-11-10 12:26:05 -08:00
Chris Lu
1dd3a6ac36
avoid verbose logs
2020-11-09 15:56:11 -08:00
Chris Lu
6856b0d57e
filer: add API to add/modify/delete tagging
2020-11-09 01:00:07 -08:00
Konstantin Lebedev
6cd6ff6962
avoid blank response for 404
2020-11-04 22:54:47 +05:00
Chris Lu
0dafcf1f5a
volume: detect max volume count changes based on disk usage
...
fix https://github.com/chrislusf/seaweedfs/issues/1594
2020-11-03 14:43:17 -08:00
Chris Lu
06ee199ef3
ensure entryViewCache is in sync with the chunks
2020-11-03 12:22:19 -08:00
Chris Lu
b132cb893f
in case some chunks modified earliar are added later
2020-11-03 12:22:19 -08:00
Chris Lu
0ae4d7bfdf
lock "addChunks()"
2020-11-03 12:22:19 -08:00
Konstantin Lebedev
e1190b3224
load S3 config from filer
...
https://github.com/chrislusf/seaweedfs/issues/1500
2020-11-03 21:45:56 +05:00
Chris Lu
d29b787220
store together with "Seaweed-" prefix
2020-11-03 00:21:10 -08:00
Chris Lu
5448781dfc
filer: Headers with "Seaweed-" prefix are stored as extended properties
2020-11-03 00:15:51 -08:00
Chris Lu
0ecff9521d
avoid []byte reuse
2020-11-02 23:27:54 -08:00
Chris Lu
031aa70da6
Revert "revert bytebufferpool, seems problematic"
...
This reverts commit e0fcab47c0
.
2020-11-02 19:11:25 -08:00
Chris Lu
7d5db73d98
go fmt
2020-11-02 14:21:07 -08:00
Chris Lu
a9662d44ac
avoid closing chunkSaveErrChan too early
2020-11-02 14:20:38 -08:00
Chris Lu
e0fcab47c0
revert bytebufferpool, seems problematic
2020-11-02 13:49:38 -08:00
Chris Lu
e1423a83dd
mount: clear possible leftover entry data in memory
2020-11-01 23:28:56 -08:00
Chris Lu
9b10d1ac33
2.08
2020-11-01 12:45:20 -08:00
Chris Lu
8750cac090
move to util.RetryWaitTime
2020-11-01 02:36:43 -08:00
Chris Lu
ef908e166b
break import cycle
2020-11-01 02:28:11 -08:00
Chris Lu
4609644658
mount: remove files that may still are open
...
related to https://github.com/chrislusf/seaweedfs/issues/1581
2020-11-01 01:33:26 -08:00
Chris Lu
df8d976bb0
refactoring
2020-11-01 01:58:48 -07:00
Chris Lu
7192a378cc
add retry to master clients also
2020-11-01 01:40:16 -07:00
Chris Lu
004aa9ca5d
only retry transport related errors
2020-11-01 01:21:44 -07:00
Chris Lu
f7b21973ac
mount: add retry for all operations with filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1589
2020-11-01 01:11:08 -07:00
Chris Lu
de86945aeb
go fmt
2020-10-31 16:45:38 -07:00
Chris Lu
306062b4e7
filer: add CORS support
2020-10-31 16:44:03 -07:00
Chris Lu
05acc33294
volume: add CORS support
2020-10-31 16:31:39 -07:00
Chris Lu
9708df47d1
ensure offset in the index file is positive
2020-10-31 15:50:01 -07:00
Chris Lu
10f9f1cc26
handle mknod requests
2020-10-30 23:51:32 -07:00
Chris Lu
8826601be1
mount: optional limit for the number of concurrent writers
2020-10-30 21:22:20 -07:00
Chris Lu
be95f68ca7
scaffold for sftpd
2020-10-30 20:57:08 -07:00
Konstantin Lebedev
d2a4ffdd60
glog
2020-10-31 03:21:54 +05:00
Konstantin Lebedev
46303c36bf
When the volume server is stopped, the master server immediately sees the deletion of volumes
2020-10-31 02:48:25 +05:00
Chris Lu
eb32af25d4
make space for the QR code
2020-10-30 10:55:31 -07:00
Chris Lu
7ecbb4b3c9
clean up
2020-10-30 10:46:31 -07:00
Kenny
9cc6575ab5
Update filer_copy.go
...
修改 weed filer.copy 上传目录时文件名变子目录的问题
2020-10-30 18:16:49 +08:00
Chris Lu
c057dd1f64
add back upload retries
2020-10-30 02:16:34 -07:00
Chris Lu
19098f2c2d
make a copy of the filer.entry for most read operations
2020-10-30 01:23:26 -07:00
Chris Lu
6135cbaa02
clear file.entry
2020-10-30 01:23:26 -07:00
Chris Lu
02dc51b1ed
properly clear out the file.entry
2020-10-29 23:31:06 -07:00
Chris Lu
6560ac6466
volume loading: trim out unreachable idx file content
...
fix https://github.com/chrislusf/seaweedfs/issues/1583
2020-10-29 22:25:23 -07:00
Chris Lu
0b68b68ec4
reload entry only when it is a hard link
...
fix https://github.com/chrislusf/seaweedfs/issues/1581
2020-10-29 16:32:45 -07:00
Chris Lu
47047516d6
Revert "increase default volume file size limit to 1024"
...
This reverts commit 0983060a
I am confused why this was changed to 1024 by myself. 1GB should be too large for most cases.
2020-10-29 15:46:29 -07:00
Chris Lu
6f8b426f4f
Merge pull request #1578 from taozix/master
...
S3 bucket list, response with uploaded storageclass.
2020-10-29 08:32:27 -07:00
ruitao.liu
22a9ea0512
adjust s3 header file.
2020-10-29 16:05:40 +08:00
Chris Lu
c6d9974fe7
server: Add option to enable/disable volume server
...
fix https://github.com/chrislusf/seaweedfs/issues/1572
2020-10-29 00:24:18 -07:00
ruitao.liu
b917be7955
S3 bucket list, response with uploaded storageclass.
2020-10-29 14:57:19 +08:00
Konstantin Lebedev
918546bbdb
add WaitForReady to grpc DialOption
2020-10-29 08:29:04 +05:00
Konstantin Lebedev
fab01f9d8d
add readonly
2020-10-28 23:17:03 +05:00
Konstantin Lebedev
884db215a1
add shell command volume mark writable
2020-10-28 22:47:09 +05:00
Konstantin Lebedev
1ad1b8c4f6
collection Volume ReadOnly Count with detailed status
2020-10-28 17:38:26 +05:00
Chris Lu
326fcdd86b
Merge pull request #1569 from taozix/master
...
save s3 metadata to filer.
2020-10-28 03:27:24 -07:00
ruitao.liu
2dcc178d0a
save s3 metadata to filer.
2020-10-28 18:16:05 +08:00
Chris Lu
744c5594da
volume: automatically trim out unreachable entries
2020-10-28 01:14:39 -07:00
Chris Lu
3e925faddd
trim out name extension
2020-10-28 01:14:05 -07:00
shibinbin
a67eb1afa1
fix: restart volumeserver sometime occurred out of memory error
2020-10-28 12:03:06 +08:00
Chris Lu
53c3aad875
volume: add a note file to avoid incomplete volume files
...
fix https://github.com/chrislusf/seaweedfs/issues/1567
2020-10-27 15:56:49 -07:00
Chris Lu
6da87720eb
shorter tag prefix
2020-10-27 15:33:26 -07:00
Chris Lu
24bf142596
copy large file first
2020-10-27 15:31:15 -07:00
Chris Lu
432c6f8db3
Merge pull request #1566 from taozix/master
...
return x-amz-tag-count header when GET object.
2020-10-27 13:16:58 -07:00
Chris Lu
06c15ab35c
volume: add special handling for .dat larger than 32GB
2020-10-27 13:11:56 -07:00
ruitao.liu
72f6b77dea
fix tag prefix.
2020-10-27 18:01:37 +08:00
ruitao.liu
8766ca1b95
rename s3 tag prefix.
2020-10-27 17:33:24 +08:00
ruitao.liu
19026ae55d
return x-amz-tag-count header when GET object.
2020-10-27 16:49:31 +08:00
Chris Lu
44921220b0
2.07
2020-10-25 22:03:46 -07:00
Chris Lu
e71463a9eb
mount: invalide file cache when metadata is changed
2020-10-25 19:24:15 -07:00
Chris Lu
e219c57849
passing full path when assign volume locations
2020-10-25 15:46:29 -07:00
Chris Lu
f375b93aef
renaming
2020-10-25 15:32:43 -07:00
Chris Lu
63b0fb54f7
minor
2020-10-25 14:15:53 -07:00
Chris Lu
27ba0a2be3
2.06
2020-10-25 12:36:47 -07:00
Chris Lu
147d4d23b6
filer.copy: automatically use the right bucket
2020-10-25 11:21:33 -07:00
Chris Lu
91fd311f7a
fix logic to read entry or not
2020-10-25 10:33:51 -07:00
Chris Lu
10a4a628e9
refresh cached file entry from sync metadata updates
2020-10-25 09:42:50 -07:00
Konstantin Lebedev
2fb1fce8a0
avoid old values when missing read only flags
2020-10-25 16:21:40 +05:00
Chris Lu
da36abf033
go fmt
2020-10-24 20:12:04 -07:00
Chris Lu
16fae84414
fix format error
2020-10-24 20:11:31 -07:00
Chris Lu
b7b1eb51b1
default to 2 for better performance in common computer hardware
2020-10-24 20:08:26 -07:00
Chris Lu
79690b9cd8
revert changes causing wrong data
2020-10-24 20:07:19 -07:00
Chris Lu
9104cfa744
reduce locks
2020-10-24 19:40:35 -07:00
Chris Lu
e9d40b80b2
less verbose logs
2020-10-24 09:42:54 -07:00
Chris Lu
6c4fb243a7
remove limits to concurrent writers
2020-10-24 02:05:05 -07:00
Chris Lu
e0002f8dd7
check existing volumes for writable status
2020-10-24 01:34:31 -07:00
Chris Lu
19772d70d7
print for debugging
2020-10-24 00:12:02 -07:00
Chris Lu
5b92dfe302
adjust log level
2020-10-23 23:58:54 -07:00
Chris Lu
fb124f2d4a
conditionally display volume server public url link
2020-10-23 23:58:46 -07:00
Chris Lu
1b17f71939
adjust election timeout to 10 seconds
2020-10-23 23:06:44 -07:00
Chris Lu
ef9c32ea0d
adjust compression threshold
2020-10-23 23:05:55 -07:00
Chris Lu
0cea84e7ab
make it easy with setting up multiple masters in a clcuster
2020-10-23 18:18:46 -07:00
Chris Lu
fc689319ae
add volume public url
2020-10-23 18:03:23 -07:00
Chris Lu
71624858af
avoid reusing cached []byte
...
this logic is just to ensure 100% correct.
fuse message, data+message hearder > 512 will not be cached.
2020-10-23 11:31:57 -07:00
Chris Lu
29c9fa2ef2
byte buffer for uploading
2020-10-23 11:29:51 -07:00
Chris Lu
2579edbc60
fix bug found by tests
2020-10-23 10:05:40 -07:00
Chris Lu
e0c8507d93
memory pool to reduce memory allocation
2020-10-23 00:41:40 -07:00
Chris Lu
b81359823f
postgres: support empty user
2020-10-22 14:27:47 -07:00
Chris Lu
11716fbf6f
make reading error more obvious
2020-10-22 13:33:45 -07:00
Chris Lu
d75bc62196
to read files that are quickly changing
2020-10-22 12:59:15 -07:00
Chris Lu
575d7952a1
add available resource stats
...
fix https://github.com/chrislusf/seaweedfs/issues/1555
2020-10-22 09:13:47 -07:00
Chris Lu
5179e559f7
skip empty logs
2020-10-22 00:35:48 -07:00
Chris Lu
de044e466e
Merge pull request #1553 from kmlebedev/multi_domainname
...
Multi domainname
2020-10-21 23:56:03 -07:00
Chris Lu
e73d6c9526
able to print partial file id
...
address https://github.com/chrislusf/seaweedfs/issues/1552
2020-10-21 23:48:07 -07:00
Konstantin Lebedev
a9a7005687
set desc of option
2020-10-22 11:23:00 +05:00
Chris Lu
720b1d9b88
adding locking to avoid nil VolumeLocationList
...
fix panic: runtime error: invalid memory address or nil pointer dereference
Oct 22 00:53:44 bedb-master1 weed[8055]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x17658da]
Oct 22 00:53:44 bedb-master1 weed[8055]: goroutine 310 [running]:
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLocationList).Length(...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_location_list.go:35
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLayout).enoughCopies(...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_layout.go:376
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLayout).ensureCorrectWritables(0xc000111d50, 0xc000b55438)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_layout.go:202 +0x5a
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*Topology).SyncDataNodeRegistration(0xc00042ac60, 0xc001454d30, 0x1, 0x1, 0xc0005fc000, 0xc00135de40, 0x4, 0xc00135de50, 0x10, 0x10d, ...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/topology.go:224 +0x616
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/server.(*MasterServer).SendHeartbeat(0xc000162700, 0x23b97c0, 0xc000ae2c90, 0x0, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/server/master_grpc_server.go:106 +0x325
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/pb/master_pb._Seaweed_SendHeartbeat_Handler(0x1f8e7c0, 0xc000162700, 0x23b0a60, 0xc00024b440, 0x3172c38, 0xc000ab7100)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/pb/master_pb/master.pb.go:4250 +0xad
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100, 0xc0001fea80, 0x311fec0, 0x0, 0x0, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1329 +0xcd8
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).handleStream(0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1409 +0xc5c
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0001ce8b0, 0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:746 +0xa5
Oct 22 00:53:44 bedb-master1 weed[8055]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:744 +0xa5
Oct 22 00:53:44 bedb-master1 systemd[1]: weedmaster.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 22 00:53:44 bedb-master1 systemd[1]: weedmaster.service: Failed with result 'exit-code'.
2020-10-21 23:15:48 -07:00
Chris Lu
2bc05cfbc8
fix typo
2020-10-21 22:26:41 -07:00
Chris Lu
4ff2ceee33
UI fix on rendering EC volumes
...
addressing UI problem with https://github.com/chrislusf/seaweedfs/issues/1551
2020-10-21 22:05:58 -07:00
Chris Lu
f5ee19e0db
shortcut for appending operations
2020-10-21 20:44:01 -07:00
Chris Lu
b5917bf846
clean up
2020-10-21 19:31:26 -07:00
Chris Lu
9884dfc369
use bytebuffer for interval list
2020-10-21 19:29:51 -07:00
Chris Lu
bd103c143a
add lock for vidCache
2020-10-21 19:28:59 -07:00
Konstantin Lebedev
05b5f12f2e
multiplate DomainNames through comma
2020-10-21 20:48:51 +05:00
Chris Lu
81cf8d04df
minor
2020-10-21 02:42:23 -07:00
Chris Lu
d90e7ee3cc
adjust log level
2020-10-21 02:17:40 -07:00
Chris Lu
3bf0116de1
mount: less channel waiting
2020-10-21 02:16:21 -07:00
Chris Lu
c31b254248
mount: shortcut when there is only one chunk
2020-10-20 23:48:29 -07:00
Chris Lu
f64252023e
Revert "a better byte buffer pool"
...
This reverts commit 59c6422777
.
2020-10-20 23:21:54 -07:00
Chris Lu
ad652ffb97
avoid close closed channel
2020-10-20 22:54:34 -07:00
Chris Lu
20442dc1bc
avoid extra data copying
2020-10-20 22:54:21 -07:00
Chris Lu
59c6422777
a better byte buffer pool
2020-10-20 22:53:58 -07:00
Chris Lu
6c92fa349d
filer: support default rack
...
fix https://github.com/chrislusf/seaweedfs/issues/1546
2020-10-20 17:41:39 -07:00
Chris Lu
b3aa2fab9a
s3: report error to s3 when updating an object but it is already a directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1545
2020-10-20 10:25:16 -07:00
Chris Lu
410b818aa7
master: avoid timer leakage
2020-10-19 14:24:57 -07:00
Chris Lu
69f0da5d73
mount: pass along replication and collection parameters
2020-10-19 00:12:42 -07:00
Chris Lu
c0ab458671
report nil instead of EOF for empty files
...
related to https://github.com/chrislusf/seaweedfs/issues/1541
2020-10-17 11:03:46 -07:00
Chris Lu
898db14729
2.05
2020-10-17 02:42:33 -07:00
Chris Lu
09bab17aff
mount: avoid "send on closed channel"
2020-10-16 12:54:23 -07:00
Chris Lu
9d80a3428c
add debug_webdav
2020-10-16 12:53:56 -07:00
Chris Lu
37f165d743
webdav: return io.EOF when at end of a file
...
fix https://github.com/chrislusf/seaweedfs/issues/1344
2020-10-16 12:53:37 -07:00
Chris Lu
68d39c86f1
mysql, postgres, cassandra: change kv key to base64 encoding
...
The exisitng key-value operation for stores using mysql, postgres, and maybe cassandra are already broken.
The kv is used to store hardlink, filer store signature and replication progress.
So users using hardlink and also uses mysql, postgres, or cassandra will have broken hard links.
Users using filer.sync will need to re-sync the files.
2020-10-16 11:10:12 -07:00
Chris Lu
ee1fc6558a
refactor
2020-10-16 11:02:44 -07:00
Chris Lu
06e6341097
set initial write time to avoid race condition with delayed chunks
2020-10-15 11:08:45 -07:00
Chris Lu
d598c47136
log errors
2020-10-15 11:01:43 -07:00
Chris Lu
5039aff310
release the goroutine to read errors
2020-10-15 10:59:18 -07:00
Chris Lu
ace0ea3d28
s3: avoid duplicated bucket
2020-10-15 10:52:20 -07:00
Konstantin Lebedev
dc2e13092d
add number of read only volumes metric
2020-10-15 15:48:40 +05:00
Chris Lu
e714c28a02
mount: async write file chunk
2020-10-14 23:28:03 -07:00
Chris Lu
c95d3d9dff
adjust tests
2020-10-14 12:27:52 -07:00
Chris Lu
93bcf56514
file read report EOF
...
fix https://github.com/chrislusf/seaweedfs/issues/1344
2020-10-14 12:18:24 -07:00
Chris Lu
1069b325dd
shell: volumeServer.evacuate adds printout for ec volumes
2020-10-13 20:26:03 -07:00
Chris Lu
0542911e29
go fmt
2020-10-13 19:50:46 -07:00
Chris Lu
58fa506491
minor
2020-10-13 19:50:22 -07:00
Chris Lu
28d4e1a51b
mount: retry for directory listing with filer
...
related to https://github.com/chrislusf/seaweedfs/issues/1530
2020-10-13 19:49:52 -07:00
Chris Lu
c127da1219
filer: linearize timeout for large chunk of data
2020-10-13 14:04:46 -07:00
Chris Lu
aac4cb1f0c
adds errror on read and write
2020-10-13 13:53:34 -07:00
Chris Lu
9b4f7fed14
mount: report filer IO error
...
related to https://github.com/chrislusf/seaweedfs/issues/1530
2020-10-13 11:21:13 -07:00
Chris Lu
3f7d1d1bf1
Only wait on retryable requests
2020-10-13 00:29:46 -07:00
Chris Lu
b18f21cce1
mount: fix bound tree with filer.path
...
fix https://github.com/chrislusf/seaweedfs/issues/1528
2020-10-12 21:58:37 -07:00
Chris Lu
f022aff289
add back http.StatusPartialContent
...
revert e7c04af1d0
2020-10-12 12:26:25 -07:00
Chris Lu
bbd0afd37e
2.04
2020-10-11 21:25:30 -07:00
Chris Lu
7704469d29
mount: outsideContainerClusterMode changed to use volume server publicUrl
2020-10-11 20:42:15 -07:00
Chris Lu
723ae11db4
refactoring in order to adjust volume server url later
2020-10-11 20:15:10 -07:00
Chris Lu
d155f907c2
mount: configurable read wait time
2020-10-10 20:09:43 -07:00
Chris Lu
9b0e8ef026
filer: added QR code to transfer files with mobile devices
2020-10-10 18:00:21 -07:00
Chris Lu
8a52379ecb
add retry if volume can not be found
2020-10-10 16:02:39 -07:00
Chris Lu
70af0ec24c
filer: fix hanging on read
2020-10-10 16:02:10 -07:00
Chris Lu
cff8bb6554
return proper error
2020-10-10 15:43:22 -07:00
Chris Lu
6da8eef54f
fix logic error
2020-10-09 00:01:47 -07:00
Chris Lu
5d01dd28a6
add logging
2020-10-08 23:38:00 -07:00
Chris Lu
8d34eb0050
mount:exponentially backoff if read error for about 10 minutes
2020-10-08 23:31:26 -07:00
Chris Lu
6e1f936efd
refactoring
2020-10-08 23:19:42 -07:00
Chris Lu
b2ee5873fb
fix error not being returned
2020-10-08 23:19:20 -07:00
Chris Lu
b067853162
mount: ignore recursion error
...
fix https://github.com/chrislusf/seaweedfs/issues/1514
the host OS should manage recursive deletion.
2020-10-08 21:55:55 -07:00
Chris Lu
e1a8e3f900
fix logging
2020-10-08 18:40:38 -07:00
Chris Lu
8676db0f6a
adjust logging
2020-10-08 18:33:37 -07:00
Chris Lu
bd8b9b0c2e
some changes to prepare fix after failed reads
2020-10-08 18:33:06 -07:00
Chris Lu
78859e35fc
s3: adjust config examples
2020-10-08 10:12:09 -07:00
Chris Lu
9e7a2772b1
s3: only admin can list all buckets
2020-10-08 10:12:09 -07:00
Chris Lu
eed492b73b
randomize file locations
2020-10-07 23:58:32 -07:00
Chris Lu
ec08a1670b
volume: avoid hanging when stopping a volume server
2020-10-07 23:48:24 -07:00
Chris Lu
62d3d3aea0
go fmt
2020-10-07 23:30:54 -07:00
Chris Lu
e91b9c85a2
s3: Added support for "List" action in weed s3 -config=... in the config file.
...
fix https://github.com/chrislusf/seaweedfs/issues/1511
2020-10-07 23:22:35 -07:00
Chris Lu
a8624c2e4f
read from alternative replica
...
related to https://github.com/chrislusf/seaweedfs/issues/1512
2020-10-07 22:49:04 -07:00
Chris Lu
da4edf3651
master: check peers for existing leader before starting a leader election
...
fix https://github.com/chrislusf/seaweedfs/issues/1509
2020-10-07 01:25:39 -07:00
Chris Lu
c543762e23
volume: auto expands -max setting
...
I have tried to run weed volume but got errors...
cmd:
weed volume -max=0 -mserver="127.0.0.1:9333" -port=8080 -dir="/var/www/d1/sfs,/var/www/d2/sfs,/var/www/d3/sfs"
log:
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d1/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d2/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d3/sfs Permission: -rwxr-xr-x
F1006 15:09:26 17040 volume.go:142] 3 directories by -dir, but only 1 max is set by -max
What am I doing wrong? Thx.
2020-10-06 09:05:30 -07:00
Chris Lu
36492c47ec
adjust
2020-10-05 14:06:18 -07:00
Chris Lu
8f8e9ddbde
adjust API
2020-10-05 09:47:07 -07:00
Konstantin Lebedev
e4f2d9eb4a
We return etag using the same algorithm as aws s3
...
https://teppen.io/2018/06/23/aws_s3_etags/
2020-10-05 14:43:32 +05:00
Chris Lu
4fc673341f
2.03
2020-10-04 21:58:22 -07:00
Chris Lu
6b591b02af
adjust log level
2020-10-04 16:21:43 -07:00
Устюжанин Антон Александрович
702b1cb876
fix: remove deleted peers if resumeState = true
2020-10-04 21:56:17 +05:00
Chris Lu
d522df0ecd
adjust log level
2020-10-04 01:35:32 -07:00
Chris Lu
666859f7cd
mount: read prefetching
2020-10-04 01:31:04 -07:00
Chris Lu
bc47835997
mount: avoid duplicated reads
2020-10-03 20:16:42 -07:00
Chris Lu
8d65ad1444
Revert "mount: adds read prefetching"
...
This reverts commit 692f0614d1
.
2020-10-03 19:40:56 -07:00
Chris Lu
692f0614d1
mount: adds read prefetching
2020-10-03 16:50:09 -07:00
Chris Lu
707936f482
re-enable caching larger than 16MB
...
revert 62ce85610e
2020-10-03 14:12:38 -07:00
Chris Lu
2c20ef72ae
fix starting problem
2020-10-03 13:56:38 -07:00
Chris Lu
d77e3c96e9
FUSE: use the mount directory permissions
2020-10-03 13:37:35 -07:00
Chris Lu
95ffed87c0
Merge pull request #1505 from ustuzhanin/remove_raft_state
...
Resume raft state
2020-10-03 10:38:47 -07:00
Устюжанин Антон Александрович
6fbfe73683
fix: resumeState
2020-10-03 22:16:47 +05:00
Устюжанин Антон Александрович
dc31b19469
fix: restore raft state
2020-10-03 14:03:41 +05:00
Chris Lu
f781cce500
s3: support object tagging
...
* GetObjectTagging
* PutObjectTagging
* DeleteObjectTagging
2020-10-02 22:21:51 -07:00
Устюжанин Антон Александрович
8c82fb7e5f
fix: restore raft state
2020-10-02 23:01:20 +05:00
Chris Lu
9ab98fa912
s3 metrics adjust the label
2020-10-01 07:17:47 -07:00
Chris Lu
f1e879fe7a
2.02
2020-10-01 07:15:49 -07:00
Chris Lu
a1c01d716b
volume: avoid deadlock when deleting volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/1501
2020-10-01 07:10:03 -07:00
Konstantin Lebedev
86329bbf2b
label name is statusCode
2020-10-01 01:22:38 +05:00
Konstantin Lebedev
68463e92c1
add status code in S3RequestCounter
2020-10-01 00:59:39 +05:00
Chris Lu
79ab10e300
adjust help message
2020-09-30 09:32:00 -07:00
Chris Lu
55cb68c09c
filer: replication follows master config if not specified
2020-09-30 09:15:55 -07:00
Chris Lu
b9887504e8
fix test
2020-09-27 23:19:50 -07:00
Chris Lu
f46eae284e
adjust for test
2020-09-27 23:08:11 -07:00
Chris Lu
75aca5e13d
2.01
2020-09-27 23:02:41 -07:00
Chris Lu
e6552b5e1e
filer: able to start s3 together
2020-09-27 23:00:43 -07:00
Chris Lu
474e2b6ac3
add a hard link marker to 16byte + maker, for future extensions
2020-09-27 22:38:30 -07:00
Chris Lu
c49e2bb9a3
adjust
2020-09-27 12:07:45 -07:00
Chris Lu
62ce85610e
skip caching too large chunks
2020-09-27 11:58:48 -07:00
Chris Lu
9ad2dcca2b
more tests
2020-09-27 11:42:51 -07:00
Chris Lu
e43d86c796
fix pre allocated volume size
2020-09-27 10:58:19 -07:00
Chris Lu
31fc7bb2e1
refactor
...
adjust for faster test
2020-09-27 10:41:29 -07:00
Chris Lu
a37535cd9f
avoid non utf-8 in filename
...
fix https://github.com/chrislusf/seaweedfs/issues/1493
2020-09-26 00:13:39 -07:00
Chris Lu
41be7a4c8f
filer: upload also set file size
...
ensure works same as fuse mount
2020-09-25 12:02:06 -07:00
Chris Lu
35f6518c36
weed upload: usePublicUrl was not being used on big files
...
fix https://github.com/chrislusf/seaweedfs/issues/1492
2020-09-25 02:41:38 -07:00
Chris Lu
22049dc2e6
Merge pull request #1491 from hilimd/master
...
s3: fix get object metadata problem
2020-09-25 02:35:00 -07:00
limd
402aef8f30
s3:
...
1.fix spark reading S3 directory wildcard problem
2.fix the problem of the spark history service writing S3 directory
reference git revsion number: b41b7ea4d0
2020-09-25 14:37:02 +08:00
Chris Lu
8516517c48
filer store: Cassandra supports username/password
2020-09-24 21:31:06 -07:00
limd
ed7816681a
Add cassandra authenticator mode
2020-09-25 11:22:47 +08:00
limd
370a98cf6b
code style
2020-09-25 11:19:42 +08:00
limd
aee3fd08b9
code style
2020-09-25 11:04:32 +08:00
Chris Lu
043b063136
Merge pull request #1482 from hilimd/master
...
Fix: s3 delete object
2020-09-24 18:21:34 -07:00
limd
48c578410f
mount: rollback default value
2020-09-25 09:18:52 +08:00
Chris Lu
dbf5327b98
s3: handle response-content-encoding
...
fix https://github.com/chrislusf/seaweedfs/issues/1487
2020-09-24 18:09:52 -07:00
Chris Lu
7726965a47
minor
2020-09-24 11:20:12 -07:00
Chris Lu
0790c6d605
fix empty hard link id
2020-09-24 11:16:43 -07:00
Chris Lu
1012df7bb5
switch hardlink id from int64 to bytes
2020-09-24 11:11:42 -07:00
Chris Lu
4856bce0ee
adjust for metrics port
2020-09-24 10:21:23 -07:00
Chris Lu
d2d3aec3e1
consolidate to one metricsPort in "weed server" mode
2020-09-24 09:55:02 -07:00
Chris Lu
2e9099369e
Merge pull request #1489 from kmlebedev/promhttp
...
Promhttp
2020-09-24 09:50:48 -07:00
Chris Lu
2e7c361a0d
hardlink deletion factors in hardlink counter
2020-09-24 09:43:52 -07:00
Chris Lu
1295347958
adjust hardlink update
...
simplify logic, pass entity content directly to hard link. The "weed mount" handles the logic to calculate hard link counter.
2020-09-24 09:43:00 -07:00
Konstantin Lebedev
98e9de6e11
fix style
2020-09-24 17:48:39 +05:00
Konstantin Lebedev
324e44d4b3
add start metrics server
2020-09-24 17:45:39 +05:00
limd
cbca14edc5
mount: fix k8s pvc and os mount directory permission bug
2020-09-24 18:07:16 +08:00
Chris Lu
5e239afdfc
hardlink works now
2020-09-24 03:06:48 -07:00
limd
59e91e9c7e
mount: fix k8s pvc mount directory permission
2020-09-24 10:25:45 +08:00
Chris Lu
c7d7b1a0f6
Merge pull request #1485 from LIBA-S/fix_oversized
...
Correct the oversized state of volume after compaction
2020-09-23 19:24:30 -07:00
Chris Lu
b80a2b3cc9
Merge pull request #1486 from LIBA-S/fix_race_condition
...
Fix a race condition when handle VolumeLocationList
2020-09-23 19:24:16 -07:00
LIBA-S
eecd6b5d35
Fix a race condition when handle VolumeLocationList
2020-09-23 20:56:51 +08:00
LIBA-S
0157798ebf
Correct the oversized state of volume after compaction
2020-09-23 20:27:42 +08:00
limd
9b21ec27e4
mount: auto created dir set corrct umask fix bug
2020-09-23 18:35:37 +08:00
hilimd
0db149fb5f
Merge pull request #18 from chrislusf/master
...
sync
2020-09-23 17:32:05 +08:00
Chris Lu
f7a0ccb595
mount: auto created dir set corrct umask
2020-09-23 02:31:19 -07:00
Chris Lu
f0e325b6fc
mount: auto created directory follow umask
2020-09-23 02:27:57 -07:00
limd
d506080c36
rollback
2020-09-23 14:29:53 +08:00
limd
8f9f29b773
fixed the problem of empty directory when S3 deleted the directory
2020-09-23 13:33:13 +08:00
limd
1892677b22
fixed the problem of empty directory when S3 deleted the directory
2020-09-23 13:15:06 +08:00
hilimd
85570f27a0
Merge pull request #17 from chrislusf/master
...
sync
2020-09-23 11:39:57 +08:00
limd
a99f63cb4d
1.add S3 copy directory function
...
2.fixed the problem of empty directory when S3 deleted the directory
2020-09-23 11:26:01 +08:00
Chris Lu
b61d33f251
expose only store wrapper to meta changes
2020-09-22 16:27:36 -07:00
Chris Lu
0adbb56cc1
rename
2020-09-22 16:24:13 -07:00
Chris Lu
3e52329cee
Revert "Merge pull request #1479 from LIBA-S/fix_oversized"
...
This reverts commit bd11f0b3e4
, reversing
changes made to ec5b9f1e91
.
2020-09-22 15:05:37 -07:00
Chris Lu
4a1fe4b8e2
add error logs
2020-09-22 09:16:10 -07:00
LIBA-S
8573ac82c6
Fix: remove the oversized state after compaction
2020-09-22 21:48:39 +08:00
James Hartig
658fc2e5b6
Allow option to enable volume pprof on server
2020-09-21 22:43:10 -04:00
James Hartig
91e4eca1e9
Fix deadlock with KeepConnected and SendHeartbeat
...
There's the potential where we're writing to a clientConn and it goes away
and we're stuck keeping a read lock on clientChansLock. This causes
KeepConnected to not be able to remove the client since it requires a write
lock on clientChansLock. This ends up backing up SendHeartbeat because it
can't get a read lock.
2020-09-21 22:41:38 -04:00
Chris Lu
9cdbfc1a49
refactor
2020-09-21 17:34:38 -07:00
Chris Lu
63373a9f9f
filer: add file extended properties
2020-09-21 11:08:34 -07:00
Chris Lu
b9c1f3e9de
s3: fixes for list multipart upload
2020-09-21 10:51:24 -07:00
Chris Lu
289e62a305
master: better locking of in memory volume data
...
related to https://github.com/chrislusf/seaweedfs/issues/1436#issuecomment-695880135
2020-09-20 23:07:55 -07:00
Chris Lu
9a3b564508
adjust printout
2020-09-20 16:01:56 -07:00
Chris Lu
62563a895a
refactoring
2020-09-20 16:00:01 -07:00
Chris Lu
c2faab23b6
refactor
2020-09-20 15:40:49 -07:00
Chris Lu
d013d09a9b
adjust logging
2020-09-20 15:38:59 -07:00
Chris Lu
f498c71199
shell: move volume operations to use flag parsing arguments
2020-09-20 09:27:34 -07:00
Chris Lu
fcbc520373
fix print out
2020-09-20 08:47:30 -07:00
Chris Lu
56094541c8
shell: adjust command option for collection.delete
2020-09-20 08:46:16 -07:00
Chris Lu
9cac9c4001
2.00
2020-09-20 00:47:04 -07:00
Chris Lu
dc4d2145ff
open up
2020-09-19 20:23:12 -07:00
Chris Lu
29abe980df
s3: add support for PostPolicy
...
fix https://github.com/chrislusf/seaweedfs/issues/1426
2020-09-19 20:14:19 -07:00
Chris Lu
41d508edfd
go fmt
2020-09-19 14:10:26 -07:00
Chris Lu
5b40a2690a
refactoring
2020-09-19 14:09:58 -07:00
Chris Lu
2c21eb1971
volume: get metrics configuration from master
...
fix https://github.com/chrislusf/seaweedfs/issues/1354
2020-09-19 00:03:00 -07:00
Chris Lu
2cbd1cf121
fix compilation
2020-09-18 00:15:54 -07:00
Chris Lu
23e9ede068
s3: collect metrics
2020-09-18 00:09:04 -07:00
Chris Lu
852e5f7cbc
filer: fix mongodb insert
...
fix https://github.com/chrislusf/seaweedfs/issues/1471
2020-09-17 21:50:52 -07:00
Chris Lu
6544e60bea
s3 add metrics
...
empty for now
2020-09-17 06:56:15 -07:00
Chris Lu
cb427d48fa
filer report metrics configuration
2020-09-17 06:46:51 -07:00
Chris Lu
e4e0234998
refactoring
2020-09-17 06:43:54 -07:00
Chris Lu
feca07bf96
Filer: ip bind
...
fix https://github.com/chrislusf/seaweedfs/issues/1470
2020-09-16 10:26:08 -07:00
Chris Lu
e861a6a3ab
simplify metrics configuration logic
2020-09-16 01:39:30 -07:00
Chris Lu
be54eeb364
passing value by checkWithMaster() instead
2020-09-16 01:33:45 -07:00
Chris Lu
4d21de63ee
go fmt
2020-09-16 01:27:24 -07:00
Chris Lu
913a16268d
volume: load configuration from master at the start
...
fix https://github.com/chrislusf/seaweedfs/issues/1469
2020-09-16 01:27:05 -07:00
Chris Lu
c9202c4b3d
add storage backend to GetMasterConfigurationResponse
2020-09-16 01:12:22 -07:00
Chris Lu
8a0710cb73
handle more than 2GB files
...
related to https://github.com/chrislusf/seaweedfs/issues/1468
2020-09-16 00:41:50 -07:00
Chris Lu
854007bc98
minor
2020-09-16 00:37:57 -07:00
Chris Lu
10f9081526
filer: adjust meta data events to received timestamp
...
if a client is already connected and start from t0. A message recieved at t+1 but with timestamp t-1 may not be processed by the client.
This commit changes to the event received time, so the replication can be ordered.
2020-09-15 01:18:33 -07:00
Chris Lu
684a875876
update help message
2020-09-15 00:40:38 -07:00
Chris Lu
2d3a904a82
shell: volumeServer.evacuate adds option to skip non moveable volumes
2020-09-15 00:33:49 -07:00
Chris Lu
a595916342
shell: add volumeServer.evacuate command
2020-09-14 23:47:11 -07:00
Chris Lu
103fafe00b
weed export: print out [start,stop) content range
2020-09-14 22:57:23 -07:00
Chris Lu
ada996fe55
filer: support createing empty folder
...
fix https://github.com/chrislusf/seaweedfs/issues/1161
2020-09-14 13:07:46 -07:00
Chris Lu
200fe5c83e
go fmt
2020-09-13 21:26:30 -07:00
Chris Lu
5d6753fb98
shell: add volumeServer.leave command
2020-09-13 21:25:51 -07:00
Chris Lu
1af95c5b76
refactoring
2020-09-13 12:41:26 -07:00
Chris Lu
f8fea19669
1.99
2020-09-12 13:46:33 -07:00
Chris Lu
b0c7de186d
filer: fix postgres prefixed directory listing problem
...
fix https://github.com/chrislusf/seaweedfs/issues/1465
2020-09-12 13:37:03 -07:00
Chris Lu
1a7afe7e6a
adjust default value
...
related to https://github.com/chrislusf/seaweedfs/issues/1453
2020-09-12 12:46:42 -07:00
Chris Lu
f2723c1bc8
do not idx file format
...
revert c9ab8d05fa
2020-09-12 12:42:36 -07:00
Chris Lu
ba984a4e29
1.98
2020-09-12 04:13:02 -07:00
Chris Lu
446e476a11
go fmt
2020-09-12 04:08:03 -07:00
Chris Lu
ea26a98753
volume: validate volume correctness if last entry is a deletion
2020-09-12 04:07:04 -07:00
Chris Lu
d15682b4a1
shell: volume.balance plan by ratio of fullness
2020-09-12 04:06:26 -07:00
Chris Lu
c0ee78d2fa
adjust make file
2020-09-12 04:05:42 -07:00
Chris Lu
cd9b89ba55
reduce default wait time to 10s before shutting down
2020-09-12 04:05:33 -07:00
Chris Lu
e2c741f76f
adjust replica placement after move
2020-09-12 01:01:19 -07:00
Chris Lu
2a0925590c
filer: etcd store avoid read with nil option
...
fix https://github.com/chrislusf/seaweedfs/issues/1463
2020-09-11 15:29:45 -07:00
Chris Lu
3984c3962f
add comment
2020-09-11 15:07:19 -07:00
Chris Lu
baa6bdf4d4
s3: listMultipartUploads fix output format
2020-09-11 15:04:01 -07:00
Chris Lu
3eda8d6dfc
s3: ListParts output xml format
...
fix https://github.com/chrislusf/seaweedfs/issues/1461
2020-09-11 14:53:50 -07:00
Chris Lu
ab201c2798
1.97
2020-09-11 13:47:50 -07:00
Chris Lu
9d4bdfcfdf
fix volume integrity checking
2020-09-11 11:34:10 -07:00
Chris Lu
1b8094ef75
weed export: export deleted files
2020-09-11 02:05:14 -07:00
Chris Lu
12a8f5294d
test for multi dc replication
2020-09-11 01:21:17 -07:00
Chris Lu
e60b2117c3
shell: volume balance follows replica placement
2020-09-11 00:29:25 -07:00
Chris Lu
89a62e8007
refactoring
2020-09-10 23:05:00 -07:00
Chris Lu
19537c9d21
1.96
2020-09-10 19:52:07 -07:00
Chris Lu
eaf9fdde99
avoid sharing context over separate goroutine
2020-09-10 19:51:03 -07:00
Chris Lu
cc5fe6f5ee
handle special characters in html link
2020-09-10 19:46:02 -07:00
Chris Lu
401ccf1509
fix test
2020-09-10 15:13:23 -07:00
Chris Lu
c9ab8d05fa
fixes for reading deleted fid
2020-09-10 14:42:52 -07:00
ruitao.liu
5b0676049a
change elastic initialize process similar as others.
2020-09-10 23:35:20 +08:00
Chris Lu
660d7c0edd
1.95
2020-09-10 01:32:05 -07:00
Chris Lu
0d9b858cfb
Merge pull request #1455 from taozix/master
...
add more basic elastic options.
2020-09-10 01:31:17 -07:00
ruitao.liu
9be4e97625
change logs print format.
2020-09-10 16:30:15 +08:00
ruitao.liu
719dc43af1
modify elastic urls from string to array.
2020-09-10 16:24:09 +08:00
ruitao.liu
72f9d7f047
use util to generate md5.
2020-09-10 16:11:18 +08:00
ruitao.liu
6a5b38c0d4
fix elastic kv ops.
2020-09-10 15:59:16 +08:00
ruitao.liu
3f7fbfddca
add more basic elastic options.
2020-09-10 14:22:07 +08:00
Chris Lu
b183ae54b5
master: changing 301 to 308 on redirect to volume servers
...
fix https://github.com/chrislusf/seaweedfs/issues/1454
2020-09-09 22:36:10 -07:00
Chris Lu
47b3f932e4
watch: adjust output format
2020-09-09 22:34:48 -07:00
Chris Lu
daf0a449f7
properly cancel context for streaming grpc
2020-09-09 12:07:15 -07:00
Chris Lu
7f69acd1f2
sync pprof
2020-09-09 11:33:52 -07:00
Chris Lu
387ab6796f
filer: cross cluster synchronization
2020-09-09 11:21:23 -07:00
Chris Lu
4fc0bd1a81
return http response directly
2020-09-09 03:53:09 -07:00
Chris Lu
5e13bc878c
adjust log level
2020-09-09 03:49:26 -07:00
Chris Lu
d1b816212f
return nil if no need to do anything
2020-09-07 16:03:05 -07:00
Chris Lu
64a621bcc8
shell: volume.fix.replication also purge over replicated volumes
2020-09-07 16:00:10 -07:00
Chris Lu
c18ea21f7a
1.94
2020-09-07 13:13:53 -07:00
Chris Lu
18b98fdb72
fix test
2020-09-07 12:57:38 -07:00
Chris Lu
44b3f2efc8
filer: Elastic Search return correct kv error
...
fix https://github.com/chrislusf/seaweedfs/issues/1452
2020-09-07 12:44:51 -07:00
Chris Lu
d80538a187
refactoring
2020-09-07 12:35:02 -07:00
Chris Lu
1a09bc43d1
refactor
2020-09-07 11:31:33 -07:00
Chris Lu
2b643f477d
typo
2020-09-06 12:47:55 -07:00
Chris Lu
05034aade5
printout over replicated locations
2020-09-06 12:44:02 -07:00
Chris Lu
432fe3d926
1.93
2020-09-06 12:12:09 -07:00
Chris Lu
d8af6f62df
adjust logging
2020-09-06 12:10:40 -07:00
Chris Lu
07f32feed0
print sync progress
2020-09-06 00:50:51 -07:00
Chris Lu
0af6252b2d
fix compilation
2020-09-06 00:50:38 -07:00
Chris Lu
ac494ff5e8
Update meta_aggregator.go
2020-09-06 00:29:16 -07:00
Chris Lu
c9f8f25ba5
read peer filer from start
2020-09-06 00:12:41 -07:00
Chris Lu
30dc365cbd
add self filer
2020-09-06 00:11:46 -07:00
Chris Lu
8acd714600
apply meta changes only if store is different
2020-09-05 23:25:03 -07:00
Chris Lu
d741ed66db
check whether shares the same filer store
2020-09-05 23:02:52 -07:00
Chris Lu
bba90ff3c8
read filer signature
2020-09-05 22:52:15 -07:00
Chris Lu
71b0e256c7
filer: setOrLoadFilerStoreSignature
2020-09-05 14:08:59 -07:00
ruitao.liu
450cf07505
skip the index that for kv usage.
2020-09-04 21:49:03 +08:00
ruitao.liu
1384ff9a2f
1.split kv in one file.
...
2.disable query for kv in es index.
2020-09-04 17:34:26 +08:00
ruitao.liu
83080b5e03
ES backended filer support kv ops.
2020-09-04 15:40:13 +08:00
Chris Lu
46f65a84a6
filer: elastic7 adjust default value
2020-09-03 20:12:38 -07:00
Chris Lu
f815fce81b
Merge pull request #1446 from taozix/master
...
new filer store option to es7.
2020-09-03 20:10:23 -07:00
Chris Lu
44b4ebf0ff
filer: Redis cleanly delete directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1448
2020-09-03 19:17:39 -07:00
Chris Lu
b8f32bcab9
filer: compress stored metadata
2020-09-03 11:00:20 -07:00
Chris Lu
f76a2b2c8a
printout meta data size
2020-09-03 09:51:21 -07:00
ruitao.liu
798280e980
change filer2 to filer.
2020-09-03 17:05:26 +08:00
ruitao.liu
a93d27d1e8
new filer option to es v7.
2020-09-03 16:34:58 +08:00
Chris Lu
1d56ea24ef
fix
2020-09-03 00:08:37 -07:00
Chris Lu
7e1aad0b54
mount: map uid/gid between local and filer
2020-09-03 00:07:22 -07:00
Chris Lu
68e878adb5
fix formatting
2020-09-02 21:42:12 -07:00
Chris Lu
0d99a5da3c
c*: support kv operations
2020-09-02 18:39:24 -07:00
Chris Lu
645a4af3db
mongodb: support kv operations
2020-09-02 17:19:14 -07:00
Chris Lu
ecaa30c408
better error message
2020-09-02 17:17:44 -07:00
Chris Lu
7c770b727c
Update abstract_sql_store_kv.go
2020-09-02 10:13:48 -07:00
Chris Lu
05a6e2dc7f
sql kv upsert
2020-09-02 10:12:44 -07:00
Chris Lu
ed62f52452
fix sql insert with duplicated primary key
2020-09-02 10:09:49 -07:00
Chris Lu
9ea290aa12
fix error message
2020-09-01 23:21:41 -07:00
Chris Lu
06a1188267
mysql/postgres: support kv operations
2020-09-01 22:48:36 -07:00
Chris Lu
aa40295f06
leveldb: report not found entry
2020-09-01 22:48:23 -07:00
Chris Lu
87d7312bf6
mysql/postgres: properly report entry not found
2020-09-01 22:47:57 -07:00
Chris Lu
72b0a5f1d1
mysql or postgres: log find error
2020-09-01 22:25:17 -07:00
Chris Lu
2815bbe6c0
go fmt
2020-09-01 21:59:26 -07:00
Chris Lu
37234bf3f8
filer store adds kv support
...
can compile now, need to implement those unimplemented
2020-09-01 21:59:00 -07:00
James Hartig
8e54e34576
volume: Don't unmount before deleting volume in copy
...
If we unmount first and then delete, the delete fails because the volume
was unmounted. Delete ends up doing the same thing as the unmount anyways.
2020-09-01 22:00:07 -04:00
Chris Lu
2b14ae5819
fix tests
2020-09-01 01:33:43 -07:00
Chris Lu
d91ec535b3
fix tests
2020-09-01 01:29:13 -07:00
Chris Lu
eb7929a971
rename filer2 to filer
2020-09-01 00:21:19 -07:00
Chris Lu
38e06d783d
volume: check disk space before compaction
...
fix https://github.com/chrislusf/seaweedfs/issues/1440
2020-08-31 18:10:53 -07:00
Chris Lu
dbb10e0f0b
testing with more than 30days
2020-08-31 17:59:29 -07:00
Chris Lu
dd176cdb8b
avoid tiny step pagination
2020-08-31 11:28:03 -07:00
Chris Lu
97733c3dff
typo
2020-08-31 10:49:17 -07:00
Chris Lu
c8398bdb09
adjust metrics
2020-08-31 10:41:05 -07:00
Chris Lu
8a1c8e41b3
simplify if else logic
2020-08-31 10:39:24 -07:00
Chris Lu
f0c89cfacd
go fmt
2020-08-31 10:23:31 -07:00
Chris Lu
edb9d65e05
Merge pull request #1431 from kmlebedev/wip-prefix-search
...
Wip prefix search
2020-08-31 10:22:14 -07:00
Konstantin Lebedev
9a195bebfd
accurate limit
2020-08-31 22:13:13 +05:00
Konstantin Lebedev
18b98f9747
not convert err to string
2020-08-31 21:55:18 +05:00
Konstantin Lebedev
22f32e75c4
rename
2020-08-31 21:52:05 +05:00
Konstantin Lebedev
a8de332beb
fix
2020-08-31 21:45:44 +05:00
Konstantin Lebedev
7f013d77d6
add var UnsupportedListDirectoryPrefixedErr
2020-08-31 21:43:32 +05:00
Konstantin Lebedev
60a86cfe04
add const
2020-08-31 21:35:16 +05:00
Konstantin Lebedev
82ea121d09
rm func
2020-08-31 18:50:06 +05:00
Konstantin Lebedev
2ea638f865
filer store wrapper can implement the logic to filter by prefi
2020-08-31 17:13:56 +05:00
Chris Lu
408e339c53
also delete the manifest chunk itself
2020-08-31 00:16:03 -07:00
Chris Lu
dd52037c27
1.92
2020-08-30 22:56:35 -07:00
Chris Lu
33d8c6c617
change log to warning
2020-08-30 21:01:44 -07:00
Chris Lu
e62d7f221f
a bit more logs
2020-08-30 21:00:24 -07:00
Chris Lu
22fe4ae573
mount: fix for UrBackup
2020-08-30 21:00:09 -07:00
Chris Lu
ae84a9ee2e
add logs
2020-08-30 20:59:04 -07:00
Chris Lu
a41588279a
change log level 5 to 4
2020-08-30 20:12:04 -07:00
Chris Lu
56244fb9a1
fix hard coded host address
2020-08-30 10:23:35 -07:00
Chris Lu
f2a8574448
filer and mount deletion resolves manifest chunks also
2020-08-30 02:07:14 -07:00
Chris Lu
99ecf63276
go fmt
2020-08-29 22:28:33 -07:00
Chris Lu
ef75ce8a34
use UTC for all time related folders
2020-08-29 21:01:14 -07:00
Chris Lu
b69cb74c03
read meta logs by timestamp
...
pass in event ts when moving logs
meta aggregator reads in memory logs only
2020-08-29 17:37:19 -07:00
Chris Lu
063c9ddac5
adjust logs
2020-08-29 11:56:22 -07:00
Chris Lu
ca658a97c5
add signatures to messages to avoid double processing
2020-08-28 23:48:48 -07:00
Chris Lu
63ad1abcce
watch: follow changes for a pattern
2020-08-28 19:43:04 -07:00
Chris Lu
b7a654a318
file mime avoid saving application/octet-stream
2020-08-28 19:42:40 -07:00
Chris Lu
707192f966
fix help message
2020-08-27 00:13:38 -07:00
Chris Lu
0be6863c87
rename
2020-08-26 22:40:15 -07:00
Chris Lu
50f7e2e89f
add log messages
2020-08-26 09:51:20 -07:00
Chris Lu
6bc618051c
log message
2020-08-26 09:32:56 -07:00
Chris Lu
9dae4e9d3e
fix logs
2020-08-26 09:31:11 -07:00
Chris Lu
4c1a3187bf
fix log error
2020-08-26 09:25:54 -07:00
Chris Lu
ab759f0ec2
erasure coding: fix EC error if multiple disks are configured in one volume server
2020-08-26 09:16:58 -07:00
Chris Lu
d1cf39f180
fix logging
2020-08-26 08:55:15 -07:00
Chris Lu
83012f543a
s: avoid possible completeMultipartUpload re-tries that can overwrite the uploaded file
2020-08-26 08:50:24 -07:00
Konstantin Lebedev
464d4c82ec
stop send heartbeat before stop volume server
2020-08-26 16:16:11 +05:00
Chris Lu
1901f15cd2
volume server: remove whitelist for status checking
...
Volume liveness/readiness not work if jwt enabled, I try use param
-whiteList
but jwt breaks
2020-08-25 02:27:47 -07:00
Chris Lu
81e5124faf
s3: list bucket permission change from admin to read
...
fix https://github.com/chrislusf/seaweedfs/issues/1430
2020-08-24 11:22:45 -07:00
Chris Lu
5f55a87101
close http response
2020-08-24 00:32:44 -07:00
Chris Lu
0b301b0b47
consume all response body in order to release requests
2020-08-23 21:32:29 -07:00
Chris Lu
2cdad77d82
fix checkin error
2020-08-23 18:30:11 -07:00
Chris Lu
81a8aa6581
1.91
2020-08-23 17:28:40 -07:00
Chris Lu
51346a5930
always keep the manifest list of chunks
2020-08-23 17:15:12 -07:00
Chris Lu
c1d1677a28
keep manifest chunks forever
2020-08-23 16:59:01 -07:00
Chris Lu
98175548c2
adjust deletion ordering
2020-08-23 16:27:12 -07:00
Chris Lu
f0e50e7b26
fix test
2020-08-23 16:12:34 -07:00
Chris Lu
aee27ccbe1
multiple fixes
...
* adjust isOpen count
* move ContinuousDirtyPages lock to filehandle
* fix problem with MergeIntoVisibles, avoid reusing slices
* let filer delete the garbage
2020-08-23 15:48:02 -07:00
Chris Lu
c19245886c
simpler test
2020-08-23 14:20:27 -07:00
Chris Lu
df816a58fe
add tests
2020-08-23 14:09:25 -07:00
Chris Lu
77393d3d30
add file handle locking when changing file entry
2020-08-23 00:35:50 -07:00
Chris Lu
d60bcbf08a
sorting chunks
2020-08-23 00:00:36 -07:00
Chris Lu
5e6b714836
add random test
2020-08-22 16:33:00 -07:00
Chris Lu
42ddbcc3a2
SQL insert falls back to update
...
fix https://github.com/chrislusf/seaweedfs/issues/1429
2020-08-20 07:52:46 -07:00
Chris Lu
f48567c5c6
remove unused function
2020-08-19 22:53:49 -07:00
Chris Lu
c45ba5d7d4
fix listObjectsV2 response format
...
fix https://github.com/chrislusf/seaweedfs/issues/1426 issue 1
2020-08-19 12:08:04 -07:00
James Hartig
3ccfa4c6ad
Added VolumeMarkWritable and VolumeStatus grpc methods
...
This is necessary for copy to mark as read-only and then restore the
original state afterwards.
2020-08-19 11:42:56 -04:00
Chris Lu
3b4b1d4a77
fix tests
2020-08-19 01:37:56 -07:00
Chris Lu
839634097f
also do flush on release
2020-08-19 01:27:10 -07:00
Chris Lu
c27e18aa6a
read possible old deleted chunks
2020-08-19 00:43:07 -07:00
Chris Lu
a78772d5ea
avoid shutdown in the middle of running
2020-08-19 00:42:02 -07:00
Chris Lu
ae9bc4a508
logs
2020-08-18 23:42:26 -07:00
Chris Lu
ed4b43b419
adjust logs
2020-08-18 23:42:09 -07:00
Chris Lu
9f1e0aeef5
delete chunks in the last step
2020-08-18 23:40:53 -07:00
Chris Lu
99d05f758c
adjust logs
2020-08-18 23:39:18 -07:00
Chris Lu
fe01191b5b
support read option readDeleted=true
2020-08-18 19:22:16 -07:00
Chris Lu
6ccd7f0a4d
refactoring
2020-08-18 18:01:37 -07:00
Chris Lu
c026eb0592
refactoring
2020-08-18 17:39:29 -07:00
Chris Lu
7e91ae592c
pass in option to read deleted entries
...
not working yet
2020-08-18 17:37:26 -07:00
Chris Lu
51ecb49db3
for debugging
2020-08-18 17:35:39 -07:00
Chris Lu
ee11d98650
refactoring
2020-08-18 17:35:19 -07:00
Chris Lu
6a92f0bc7a
refactoring to typed Size
...
Go is amazing with refactoring!
2020-08-18 17:04:28 -07:00
Chris Lu
618b2f6829
release resources only when needed to
2020-08-18 12:53:08 -07:00
Chris Lu
208849702d
logs
2020-08-18 12:52:54 -07:00
Chris Lu
1fcd083db3
printout data size
2020-08-18 09:09:29 -07:00
Chris Lu
3e5339337a
minor
2020-08-18 08:50:14 -07:00
Chris Lu
85001cbec7
properly report io.EOF
2020-08-18 08:18:54 -07:00
Chris Lu
0ca45a5cbc
adjust logs
2020-08-18 00:34:15 -07:00
Chris Lu
cd43738245
fix reading when filling zeros
2020-08-18 00:32:01 -07:00
Chris Lu
ecb3ce46be
adjust error logs
2020-08-18 00:31:42 -07:00
Chris Lu
30fe424469
add one more test case
2020-08-17 22:47:27 -07:00
Chris Lu
1b68ba953b
fix for out of range reads
2020-08-17 22:46:32 -07:00
Chris Lu
56fbd2c211
fix reading
2020-08-17 21:17:32 -07:00
Chris Lu
09e126bae5
refactoring: use interface
2020-08-17 20:20:08 -07:00
Chris Lu
be4d42b8e2
rename
2020-08-17 20:15:53 -07:00
Chris Lu
0625e63648
count 0 as part of the reads
2020-08-17 20:14:40 -07:00
Chris Lu
0be4b6e7f0
logs
2020-08-17 16:05:40 -07:00
Chris Lu
97e54a80d4
rename variables
2020-08-17 16:05:13 -07:00
Chris Lu
abdaf9958d
possibly read more
2020-08-17 16:04:56 -07:00
Chris Lu
9d46c7bc78
rename
2020-08-17 11:12:10 -07:00
Chris Lu
4ccfdaeb4d
prevent nil
2020-08-17 10:07:34 -07:00
Chris Lu
24c8e6bcb4
minor optimization
2020-08-17 10:03:34 -07:00
Chris Lu
f5837b7000
report error first
2020-08-16 23:49:10 -07:00
Chris Lu
2ac27616bc
fix possible out of range bytes
...
avoid buff out of range resp.Data = buff[:totalRead]
2020-08-16 23:47:34 -07:00
Chris Lu
ff200398bb
1.90
2020-08-16 21:18:50 -07:00
Chris Lu
4ceeba9e70
streaming reads
2020-08-16 21:07:46 -07:00
Chris Lu
e72953dff7
logs
2020-08-16 21:06:03 -07:00
Chris Lu
627b081b67
adjust logs
2020-08-16 16:32:22 -07:00
Chris Lu
22e5132b3a
adjust log level
2020-08-16 16:25:11 -07:00
Chris Lu
ee0f92a6be
reduce memory allocation
2020-08-16 16:24:40 -07:00
Chris Lu
2ba817afac
read randomly written data
2020-08-16 15:16:46 -07:00
Chris Lu
6111b265e7
fix compilation
2020-08-16 01:38:16 -07:00
Chris Lu
4a77f0820a
clean up logs
2020-08-16 01:37:50 -07:00
Chris Lu
8c9e6eaacd
fix tests
2020-08-16 01:35:52 -07:00
Chris Lu
b71df82292
1.89
2020-08-16 00:57:28 -07:00
Chris Lu
1d9ea30b72
fix ViewFromVisibleIntervals
2020-08-16 00:49:26 -07:00
Chris Lu
aec7f32b02
fix reader_at
2020-08-16 00:49:08 -07:00
Chris Lu
5d80fc2ec7
adjust logs
2020-08-15 21:09:31 -07:00
Chris Lu
e0bfd3161a
update metadata only if changed
2020-08-15 19:56:24 -07:00
Chris Lu
003d48da21
adjust logs
2020-08-15 19:55:28 -07:00
Chris Lu
6ee8d952d2
adjust log level
2020-08-15 18:24:35 -07:00
Chris Lu
3e1395b767
adjust log message
2020-08-15 17:06:16 -07:00
Chris Lu
a22ee30596
fix nil
2020-08-15 17:01:42 -07:00
Chris Lu
0d60e67816
ensure meta data changes are updated
2020-08-15 14:15:07 -07:00
Chris Lu
5b43bddf20
proper deletion ordering
...
delete central file store first, then delete local cache
2020-08-15 09:33:41 -07:00
Chris Lu
c647deace1
file size support set file length
...
use Attr.FileSize and TotalChunkSize to determine file size
2020-08-15 09:32:47 -07:00
Chris Lu
c03bb180eb
fix error reporting
2020-08-14 00:44:02 -07:00
Chris Lu
eb493283dd
deletion error report
2020-08-14 00:23:01 -07:00
Chris Lu
a7f669044e
rename also applies to open file handle
2020-08-14 00:22:49 -07:00
Chris Lu
edfa73782f
adjust log level
2020-08-14 00:22:21 -07:00
Chris Lu
0983060a90
increase default volume file size limit to 1024
...
avoid possible large chunk size set on mount or filer
2020-08-13 09:07:22 -07:00
Chris Lu
090612492c
fix compilation
2020-08-13 00:07:56 -07:00
Chris Lu
d43129d27f
add back handles lock
2020-08-12 23:52:13 -07:00
Chris Lu
f735d579d3
adjust comment
2020-08-12 13:22:19 -07:00
Chris Lu
8824a9755c
remove directory cache
2020-08-12 13:11:04 -07:00
Chris Lu
83cad3da79
add retry file upload
2020-08-11 20:30:11 -07:00
Chris Lu
152a6cbc2b
minor adjustments
2020-08-10 20:42:27 -07:00
Chris Lu
f1b40567e3
Merge pull request #1419 from robin1900/master
...
fix bug: two same volumeId in different collections
2020-08-10 20:38:55 -07:00
cheng.li01
25fbff5d52
fix bug: two same volumeId in different collections
...
1, there will be two leader when master server startup in a few seconds
2, raft server will get a leader even there is only one master, so there is no need to do hard code to set the server to be leader
2020-08-10 16:37:47 +08:00
Chris Lu
e74dc4e4bc
add back fs node cache for renaming
2020-08-09 21:56:09 -07:00
Chris Lu
f86c7d911a
remove println
2020-08-09 14:42:25 -07:00
Chris Lu
9ecc1170a3
istObjects和listObjectsV2不能查询子目录 #1418
...
fix https://github.com/chrislusf/seaweedfs/issues/1418
2020-08-09 14:35:53 -07:00
Chris Lu
3f4aff5dde
s3: fix delimiter in list response
2020-08-09 09:09:35 -07:00
Chris Lu
b056707770
1.88
2020-08-08 22:55:12 -07:00
Chris Lu
9832653e1d
FUSE mount: proper error with deleting non empty folder
2020-08-08 21:37:36 -07:00
Chris Lu
3b1a95ac26
filer refactoring: same auto chunking logic for POST and PUT, no size limit
2020-08-08 12:02:06 -07:00
Chris Lu
67348e7b15
less noisy heartbeat logs
2020-08-08 10:53:35 -07:00
Chris Lu
bee0d7e5eb
lower log priority for noisy heartbeat
2020-08-08 10:52:13 -07:00
Chris Lu
ae00cce4bd
support POST and PUT auto chunking
2020-08-08 10:45:37 -07:00
Chris Lu
bd8bfdae07
refactoring
2020-08-08 10:18:43 -07:00
Chris Lu
ab6e5c0dc4
adjust error message
2020-08-08 10:18:32 -07:00
Chris Lu
828a5ae429
check signature only when auth is enabled
2020-08-08 09:11:40 -07:00
Chris Lu
20e2ac1add
filer: store md5 metadata for files uploaded by filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1412
2020-08-06 10:04:17 -07:00
Chris Lu
93ea0801ea
volume: the variable for the master node may be stale?
...
related to https://github.com/chrislusf/seaweedfs/issues/1414
2020-08-06 09:48:54 -07:00
Chris Lu
4ecfa9879d
volume: report Content-MD5 in response header
2020-08-06 05:22:53 -07:00
Chris Lu
41007ced77
remove logging
2020-08-06 04:26:29 -07:00
Chris Lu
2b74abf766
S3: configurable access for anonymous user
...
fix https://github.com/chrislusf/seaweedfs/issues/1413
2020-08-06 03:41:34 -07:00
Устюжанин Антон Александрович
9ca011e3ca
ListDirectoryPrefixedEntries
2020-08-06 00:38:59 +05:00
Устюжанин Антон Александрович
b231f7bdab
ListDirectoryPrefixedEntries
2020-08-06 00:37:42 +05:00
Устюжанин Антон Александрович
a457c308ad
test ListDirectoryPrefixedEntries
2020-08-06 00:24:31 +05:00
Устюжанин Антон Александрович
2d3b355fb6
test ListDirectoryPrefixedEntries
2020-08-05 23:56:06 +05:00
Устюжанин Антон Александрович
67f134ebd7
test ListDirectoryPrefixedEntries
2020-08-05 23:44:48 +05:00
Устюжанин Антон Александрович
dc9fc01254
test ListDirectoryPrefixedEntries
2020-08-05 23:38:00 +05:00
Устюжанин Антон Александрович
33a9e5e2d1
test ListDirectoryPrefixedEntries
2020-08-05 22:19:16 +05:00
Chris Lu
cbd80253e3
better needle id format
2020-08-04 22:09:07 -07:00
Chris Lu
1b3a80dd3d
non-fatal error
2020-08-01 13:46:52 -07:00
Chris Lu
bbbea8159c
http request use gzip if possible
2020-08-01 13:20:52 -07:00