mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-11-30 15:19:01 +08:00
Code Issues Packages Projects Releases Wiki Activity
4,818 Commits 32 Branches 296 Tags 148 MiB
0e016bc7bd
Commit Graph

3019 Commits

Author SHA1 Message Date
Chris Lu
de3bdd0651 delete old volume replica 
related to https://github.com/chrislusf/seaweedfs/issues/1607

old is:
* older compaction revision
* older modified time
* smaller volume size
 2020-11-10 12:26:05 -08:00
Chris Lu
1dd3a6ac36 avoid verbose logs 2020-11-09 15:56:11 -08:00
Chris Lu
6856b0d57e filer: add API to add/modify/delete tagging 2020-11-09 01:00:07 -08:00
Konstantin Lebedev
6cd6ff6962 avoid blank response for 404 2020-11-04 22:54:47 +05:00
Chris Lu
0dafcf1f5a volume: detect max volume count changes based on disk usage 
fix https://github.com/chrislusf/seaweedfs/issues/1594
 2020-11-03 14:43:17 -08:00
Chris Lu
06ee199ef3 ensure entryViewCache is in sync with the chunks 2020-11-03 12:22:19 -08:00
Chris Lu
b132cb893f in case some chunks modified earliar are added later 2020-11-03 12:22:19 -08:00
Chris Lu
0ae4d7bfdf lock "addChunks()" 2020-11-03 12:22:19 -08:00
Konstantin Lebedev
e1190b3224 load S3 config from filer 
https://github.com/chrislusf/seaweedfs/issues/1500
 2020-11-03 21:45:56 +05:00
Chris Lu
d29b787220 store together with "Seaweed-" prefix 2020-11-03 00:21:10 -08:00
Chris Lu
5448781dfc filer: Headers with "Seaweed-" prefix are stored as extended properties 2020-11-03 00:15:51 -08:00
Chris Lu
0ecff9521d avoid []byte reuse 2020-11-02 23:27:54 -08:00
Chris Lu
031aa70da6 Revert "revert bytebufferpool, seems problematic" 
This reverts commit e0fcab47c0.
 2020-11-02 19:11:25 -08:00
Chris Lu
7d5db73d98 go fmt 2020-11-02 14:21:07 -08:00
Chris Lu
a9662d44ac avoid closing chunkSaveErrChan too early 2020-11-02 14:20:38 -08:00
Chris Lu
e0fcab47c0 revert bytebufferpool, seems problematic 2020-11-02 13:49:38 -08:00
Chris Lu
e1423a83dd mount: clear possible leftover entry data in memory 2020-11-01 23:28:56 -08:00
Chris Lu
9b10d1ac33 2.08 2020-11-01 12:45:20 -08:00
Chris Lu
8750cac090 move to util.RetryWaitTime 2020-11-01 02:36:43 -08:00
Chris Lu
ef908e166b break import cycle 2020-11-01 02:28:11 -08:00
Chris Lu
4609644658 mount: remove files that may still are open 
related to https://github.com/chrislusf/seaweedfs/issues/1581
 2020-11-01 01:33:26 -08:00
Chris Lu
df8d976bb0 refactoring 2020-11-01 01:58:48 -07:00
Chris Lu
7192a378cc add retry to master clients also 2020-11-01 01:40:16 -07:00
Chris Lu
004aa9ca5d only retry transport related errors 2020-11-01 01:21:44 -07:00
Chris Lu
f7b21973ac mount: add retry for all operations with filer 
fix https://github.com/chrislusf/seaweedfs/issues/1589
 2020-11-01 01:11:08 -07:00
Chris Lu
de86945aeb go fmt 2020-10-31 16:45:38 -07:00
Chris Lu
306062b4e7 filer: add CORS support 2020-10-31 16:44:03 -07:00
Chris Lu
05acc33294 volume: add CORS support 2020-10-31 16:31:39 -07:00
Chris Lu
9708df47d1 ensure offset in the index file is positive 2020-10-31 15:50:01 -07:00
Chris Lu
10f9f1cc26 handle mknod requests 2020-10-30 23:51:32 -07:00
Chris Lu
8826601be1 mount: optional limit for the number of concurrent writers 2020-10-30 21:22:20 -07:00
Chris Lu
be95f68ca7 scaffold for sftpd 2020-10-30 20:57:08 -07:00
Konstantin Lebedev
d2a4ffdd60 glog 2020-10-31 03:21:54 +05:00
Konstantin Lebedev
46303c36bf When the volume server is stopped, the master server immediately sees the deletion of volumes 2020-10-31 02:48:25 +05:00
Chris Lu
eb32af25d4 make space for the QR code 2020-10-30 10:55:31 -07:00
Chris Lu
7ecbb4b3c9 clean up 2020-10-30 10:46:31 -07:00
Kenny
9cc6575ab5
Update filer_copy.go 
修改 weed filer.copy 上传目录时文件名变子目录的问题
 2020-10-30 18:16:49 +08:00
Chris Lu
c057dd1f64 add back upload retries 2020-10-30 02:16:34 -07:00
Chris Lu
19098f2c2d make a copy of the filer.entry for most read operations 2020-10-30 01:23:26 -07:00
Chris Lu
6135cbaa02 clear file.entry 2020-10-30 01:23:26 -07:00
Chris Lu
02dc51b1ed properly clear out the file.entry 2020-10-29 23:31:06 -07:00
Chris Lu
6560ac6466 volume loading: trim out unreachable idx file content 
fix https://github.com/chrislusf/seaweedfs/issues/1583
 2020-10-29 22:25:23 -07:00
Chris Lu
0b68b68ec4 reload entry only when it is a hard link 
fix https://github.com/chrislusf/seaweedfs/issues/1581
 2020-10-29 16:32:45 -07:00
Chris Lu
47047516d6 Revert "increase default volume file size limit to 1024" 
This reverts commit 0983060a

I am confused why this was changed to 1024 by myself. 1GB should be too large for most cases.
 2020-10-29 15:46:29 -07:00
Chris Lu
6f8b426f4f
Merge pull request #1578 from taozix/master 
S3 bucket list, response with uploaded storageclass.
 2020-10-29 08:32:27 -07:00
ruitao.liu
22a9ea0512 adjust s3 header file. 2020-10-29 16:05:40 +08:00
Chris Lu
c6d9974fe7 server: Add option to enable/disable volume server 
fix https://github.com/chrislusf/seaweedfs/issues/1572
 2020-10-29 00:24:18 -07:00
ruitao.liu
b917be7955 S3 bucket list, response with uploaded storageclass. 2020-10-29 14:57:19 +08:00
Konstantin Lebedev
918546bbdb add WaitForReady to grpc DialOption 2020-10-29 08:29:04 +05:00
Konstantin Lebedev
fab01f9d8d add readonly 2020-10-28 23:17:03 +05:00
Konstantin Lebedev
884db215a1 add shell command volume mark writable 2020-10-28 22:47:09 +05:00
Konstantin Lebedev
1ad1b8c4f6 collection Volume ReadOnly Count with detailed status 2020-10-28 17:38:26 +05:00
Chris Lu
326fcdd86b
Merge pull request #1569 from taozix/master 
save s3 metadata to filer.
 2020-10-28 03:27:24 -07:00
ruitao.liu
2dcc178d0a save s3 metadata to filer. 2020-10-28 18:16:05 +08:00
Chris Lu
744c5594da volume: automatically trim out unreachable entries 2020-10-28 01:14:39 -07:00
Chris Lu
3e925faddd trim out name extension 2020-10-28 01:14:05 -07:00
shibinbin
a67eb1afa1 fix: restart volumeserver sometime occurred out of memory error 2020-10-28 12:03:06 +08:00
Chris Lu
53c3aad875 volume: add a note file to avoid incomplete volume files 
fix https://github.com/chrislusf/seaweedfs/issues/1567
 2020-10-27 15:56:49 -07:00
Chris Lu
6da87720eb shorter tag prefix 2020-10-27 15:33:26 -07:00
Chris Lu
24bf142596 copy large file first 2020-10-27 15:31:15 -07:00
Chris Lu
432c6f8db3
Merge pull request #1566 from taozix/master 
return x-amz-tag-count header when GET object.
 2020-10-27 13:16:58 -07:00
Chris Lu
06c15ab35c volume: add special handling for .dat larger than 32GB 2020-10-27 13:11:56 -07:00
ruitao.liu
72f6b77dea fix tag prefix. 2020-10-27 18:01:37 +08:00
ruitao.liu
8766ca1b95 rename s3 tag prefix. 2020-10-27 17:33:24 +08:00
ruitao.liu
19026ae55d return x-amz-tag-count header when GET object. 2020-10-27 16:49:31 +08:00
Chris Lu
44921220b0 2.07 2020-10-25 22:03:46 -07:00
Chris Lu
e71463a9eb mount: invalide file cache when metadata is changed 2020-10-25 19:24:15 -07:00
Chris Lu
e219c57849 passing full path when assign volume locations 2020-10-25 15:46:29 -07:00
Chris Lu
f375b93aef renaming 2020-10-25 15:32:43 -07:00
Chris Lu
63b0fb54f7 minor 2020-10-25 14:15:53 -07:00
Chris Lu
27ba0a2be3 2.06 2020-10-25 12:36:47 -07:00
Chris Lu
147d4d23b6 filer.copy: automatically use the right bucket 2020-10-25 11:21:33 -07:00
Chris Lu
91fd311f7a fix logic to read entry or not 2020-10-25 10:33:51 -07:00
Chris Lu
10a4a628e9 refresh cached file entry from sync metadata updates 2020-10-25 09:42:50 -07:00
Konstantin Lebedev
2fb1fce8a0 avoid old values when missing read only flags 2020-10-25 16:21:40 +05:00
Chris Lu
da36abf033 go fmt 2020-10-24 20:12:04 -07:00
Chris Lu
16fae84414 fix format error 2020-10-24 20:11:31 -07:00
Chris Lu
b7b1eb51b1 default to 2 for better performance in common computer hardware 2020-10-24 20:08:26 -07:00
Chris Lu
79690b9cd8 revert changes causing wrong data 2020-10-24 20:07:19 -07:00
Chris Lu
9104cfa744 reduce locks 2020-10-24 19:40:35 -07:00
Chris Lu
e9d40b80b2 less verbose logs 2020-10-24 09:42:54 -07:00
Chris Lu
6c4fb243a7 remove limits to concurrent writers 2020-10-24 02:05:05 -07:00
Chris Lu
e0002f8dd7 check existing volumes for writable status 2020-10-24 01:34:31 -07:00
Chris Lu
19772d70d7 print for debugging 2020-10-24 00:12:02 -07:00
Chris Lu
5b92dfe302 adjust log level 2020-10-23 23:58:54 -07:00
Chris Lu
fb124f2d4a conditionally display volume server public url link 2020-10-23 23:58:46 -07:00
Chris Lu
1b17f71939 adjust election timeout to 10 seconds 2020-10-23 23:06:44 -07:00
Chris Lu
ef9c32ea0d adjust compression threshold 2020-10-23 23:05:55 -07:00
Chris Lu
0cea84e7ab make it easy with setting up multiple masters in a clcuster 2020-10-23 18:18:46 -07:00
Chris Lu
fc689319ae add volume public url 2020-10-23 18:03:23 -07:00
Chris Lu
71624858af avoid reusing cached []byte 
this logic is just to ensure 100% correct.

fuse message, data+message hearder > 512 will not be cached.
 2020-10-23 11:31:57 -07:00
Chris Lu
29c9fa2ef2 byte buffer for uploading 2020-10-23 11:29:51 -07:00
Chris Lu
2579edbc60 fix bug found by tests 2020-10-23 10:05:40 -07:00
Chris Lu
e0c8507d93 memory pool to reduce memory allocation 2020-10-23 00:41:40 -07:00
Chris Lu
b81359823f postgres: support empty user 2020-10-22 14:27:47 -07:00
Chris Lu
11716fbf6f make reading error more obvious 2020-10-22 13:33:45 -07:00
Chris Lu
d75bc62196 to read files that are quickly changing 2020-10-22 12:59:15 -07:00
Chris Lu
575d7952a1 add available resource stats 
fix https://github.com/chrislusf/seaweedfs/issues/1555
 2020-10-22 09:13:47 -07:00
Chris Lu
5179e559f7 skip empty logs 2020-10-22 00:35:48 -07:00
Chris Lu
de044e466e
Merge pull request #1553 from kmlebedev/multi_domainname 
Multi domainname
 2020-10-21 23:56:03 -07:00
Chris Lu
e73d6c9526 able to print partial file id 
address https://github.com/chrislusf/seaweedfs/issues/1552
 2020-10-21 23:48:07 -07:00
Konstantin Lebedev
a9a7005687 set desc of option 2020-10-22 11:23:00 +05:00
Chris Lu
720b1d9b88 adding locking to avoid nil VolumeLocationList 
fix panic: runtime error: invalid memory address or nil pointer dereference
Oct 22 00:53:44 bedb-master1 weed[8055]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x17658da]
Oct 22 00:53:44 bedb-master1 weed[8055]: goroutine 310 [running]:
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLocationList).Length(...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_location_list.go:35
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLayout).enoughCopies(...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_layout.go:376
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLayout).ensureCorrectWritables(0xc000111d50, 0xc000b55438)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_layout.go:202 +0x5a
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*Topology).SyncDataNodeRegistration(0xc00042ac60, 0xc001454d30, 0x1, 0x1, 0xc0005fc000, 0xc00135de40, 0x4, 0xc00135de50, 0x10, 0x10d, ...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/topology.go:224 +0x616
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/server.(*MasterServer).SendHeartbeat(0xc000162700, 0x23b97c0, 0xc000ae2c90, 0x0, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/server/master_grpc_server.go:106 +0x325
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/pb/master_pb._Seaweed_SendHeartbeat_Handler(0x1f8e7c0, 0xc000162700, 0x23b0a60, 0xc00024b440, 0x3172c38, 0xc000ab7100)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/pb/master_pb/master.pb.go:4250 +0xad
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100, 0xc0001fea80, 0x311fec0, 0x0, 0x0, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1329 +0xcd8
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).handleStream(0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1409 +0xc5c
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0001ce8b0, 0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:746 +0xa5
Oct 22 00:53:44 bedb-master1 weed[8055]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:744 +0xa5
Oct 22 00:53:44 bedb-master1 systemd[1]: weedmaster.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 22 00:53:44 bedb-master1 systemd[1]: weedmaster.service: Failed with result 'exit-code'.
 2020-10-21 23:15:48 -07:00
Chris Lu
2bc05cfbc8 fix typo 2020-10-21 22:26:41 -07:00
Chris Lu
4ff2ceee33 UI fix on rendering EC volumes 
addressing UI problem with https://github.com/chrislusf/seaweedfs/issues/1551
 2020-10-21 22:05:58 -07:00
Chris Lu
f5ee19e0db shortcut for appending operations 2020-10-21 20:44:01 -07:00
Chris Lu
b5917bf846 clean up 2020-10-21 19:31:26 -07:00
Chris Lu
9884dfc369 use bytebuffer for interval list 2020-10-21 19:29:51 -07:00
Chris Lu
bd103c143a add lock for vidCache 2020-10-21 19:28:59 -07:00
Konstantin Lebedev
05b5f12f2e multiplate DomainNames through comma 2020-10-21 20:48:51 +05:00
Chris Lu
81cf8d04df minor 2020-10-21 02:42:23 -07:00
Chris Lu
d90e7ee3cc adjust log level 2020-10-21 02:17:40 -07:00
Chris Lu
3bf0116de1 mount: less channel waiting 2020-10-21 02:16:21 -07:00
Chris Lu
c31b254248 mount: shortcut when there is only one chunk 2020-10-20 23:48:29 -07:00
Chris Lu
f64252023e Revert "a better byte buffer pool" 
This reverts commit 59c6422777.
 2020-10-20 23:21:54 -07:00
Chris Lu
ad652ffb97 avoid close closed channel 2020-10-20 22:54:34 -07:00
Chris Lu
20442dc1bc avoid extra data copying 2020-10-20 22:54:21 -07:00
Chris Lu
59c6422777 a better byte buffer pool 2020-10-20 22:53:58 -07:00
Chris Lu
6c92fa349d filer: support default rack 
fix https://github.com/chrislusf/seaweedfs/issues/1546
 2020-10-20 17:41:39 -07:00
Chris Lu
b3aa2fab9a s3: report error to s3 when updating an object but it is already a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1545
 2020-10-20 10:25:16 -07:00
Chris Lu
410b818aa7 master: avoid timer leakage 2020-10-19 14:24:57 -07:00
Chris Lu
69f0da5d73 mount: pass along replication and collection parameters 2020-10-19 00:12:42 -07:00
Chris Lu
c0ab458671 report nil instead of EOF for empty files 
related to https://github.com/chrislusf/seaweedfs/issues/1541
 2020-10-17 11:03:46 -07:00
Chris Lu
898db14729 2.05 2020-10-17 02:42:33 -07:00
Chris Lu
09bab17aff mount: avoid "send on closed channel" 2020-10-16 12:54:23 -07:00
Chris Lu
9d80a3428c add debug_webdav 2020-10-16 12:53:56 -07:00
Chris Lu
37f165d743 webdav: return io.EOF when at end of a file 
fix https://github.com/chrislusf/seaweedfs/issues/1344
 2020-10-16 12:53:37 -07:00
Chris Lu
68d39c86f1 mysql, postgres, cassandra: change kv key to base64 encoding 
The exisitng key-value operation for stores using mysql, postgres, and maybe cassandra are already broken.

The kv is used to store hardlink, filer store signature and replication progress.

So users using hardlink and also uses mysql, postgres, or cassandra will have broken hard links.

Users using filer.sync will need to re-sync the files.
 2020-10-16 11:10:12 -07:00
Chris Lu
ee1fc6558a refactor 2020-10-16 11:02:44 -07:00
Chris Lu
06e6341097 set initial write time to avoid race condition with delayed chunks 2020-10-15 11:08:45 -07:00
Chris Lu
d598c47136 log errors 2020-10-15 11:01:43 -07:00
Chris Lu
5039aff310 release the goroutine to read errors 2020-10-15 10:59:18 -07:00
Chris Lu
ace0ea3d28 s3: avoid duplicated bucket 2020-10-15 10:52:20 -07:00
Konstantin Lebedev
dc2e13092d add number of read only volumes metric 2020-10-15 15:48:40 +05:00
Chris Lu
e714c28a02 mount: async write file chunk 2020-10-14 23:28:03 -07:00
Chris Lu
c95d3d9dff adjust tests 2020-10-14 12:27:52 -07:00
Chris Lu
93bcf56514 file read report EOF 
fix https://github.com/chrislusf/seaweedfs/issues/1344
 2020-10-14 12:18:24 -07:00
Chris Lu
1069b325dd shell: volumeServer.evacuate adds printout for ec volumes 2020-10-13 20:26:03 -07:00
Chris Lu
0542911e29 go fmt 2020-10-13 19:50:46 -07:00
Chris Lu
58fa506491 minor 2020-10-13 19:50:22 -07:00
Chris Lu
28d4e1a51b mount: retry for directory listing with filer 
related to https://github.com/chrislusf/seaweedfs/issues/1530
 2020-10-13 19:49:52 -07:00
Chris Lu
c127da1219 filer: linearize timeout for large chunk of data 2020-10-13 14:04:46 -07:00
Chris Lu
aac4cb1f0c adds errror on read and write 2020-10-13 13:53:34 -07:00
Chris Lu
9b4f7fed14 mount: report filer IO error 
related to https://github.com/chrislusf/seaweedfs/issues/1530
 2020-10-13 11:21:13 -07:00
Chris Lu
3f7d1d1bf1 Only wait on retryable requests 2020-10-13 00:29:46 -07:00
Chris Lu
b18f21cce1 mount: fix bound tree with filer.path 
fix https://github.com/chrislusf/seaweedfs/issues/1528
 2020-10-12 21:58:37 -07:00
Chris Lu
f022aff289 add back http.StatusPartialContent 
revert e7c04af1d0
 2020-10-12 12:26:25 -07:00
Chris Lu
bbd0afd37e 2.04 2020-10-11 21:25:30 -07:00
Chris Lu
7704469d29 mount: outsideContainerClusterMode changed to use volume server publicUrl 2020-10-11 20:42:15 -07:00
Chris Lu
723ae11db4 refactoring in order to adjust volume server url later 2020-10-11 20:15:10 -07:00
Chris Lu
d155f907c2 mount: configurable read wait time 2020-10-10 20:09:43 -07:00
Chris Lu
9b0e8ef026 filer: added QR code to transfer files with mobile devices 2020-10-10 18:00:21 -07:00
Chris Lu
8a52379ecb add retry if volume can not be found 2020-10-10 16:02:39 -07:00
Chris Lu
70af0ec24c filer: fix hanging on read 2020-10-10 16:02:10 -07:00
Chris Lu
cff8bb6554 return proper error 2020-10-10 15:43:22 -07:00
Chris Lu
6da8eef54f fix logic error 2020-10-09 00:01:47 -07:00
Chris Lu
5d01dd28a6 add logging 2020-10-08 23:38:00 -07:00
Chris Lu
8d34eb0050 mount:exponentially backoff if read error for about 10 minutes 2020-10-08 23:31:26 -07:00
Chris Lu
6e1f936efd refactoring 2020-10-08 23:19:42 -07:00
Chris Lu
b2ee5873fb fix error not being returned 2020-10-08 23:19:20 -07:00
Chris Lu
b067853162 mount: ignore recursion error 
fix https://github.com/chrislusf/seaweedfs/issues/1514

the host OS should manage recursive deletion.
 2020-10-08 21:55:55 -07:00
Chris Lu
e1a8e3f900 fix logging 2020-10-08 18:40:38 -07:00
Chris Lu
8676db0f6a adjust logging 2020-10-08 18:33:37 -07:00
Chris Lu
bd8b9b0c2e some changes to prepare fix after failed reads 2020-10-08 18:33:06 -07:00
Chris Lu
78859e35fc s3: adjust config examples 2020-10-08 10:12:09 -07:00
Chris Lu
9e7a2772b1 s3: only admin can list all buckets 2020-10-08 10:12:09 -07:00
Chris Lu
eed492b73b randomize file locations 2020-10-07 23:58:32 -07:00
Chris Lu
ec08a1670b volume: avoid hanging when stopping a volume server 2020-10-07 23:48:24 -07:00
Chris Lu
62d3d3aea0 go fmt 2020-10-07 23:30:54 -07:00
Chris Lu
e91b9c85a2 s3: Added support for "List" action in weed s3 -config=... in the config file. 
fix https://github.com/chrislusf/seaweedfs/issues/1511
 2020-10-07 23:22:35 -07:00
Chris Lu
a8624c2e4f read from alternative replica 
related to https://github.com/chrislusf/seaweedfs/issues/1512
 2020-10-07 22:49:04 -07:00
Chris Lu
da4edf3651 master: check peers for existing leader before starting a leader election 
fix https://github.com/chrislusf/seaweedfs/issues/1509
 2020-10-07 01:25:39 -07:00
Chris Lu
c543762e23 volume: auto expands -max setting 
I have tried to run weed volume but got errors...
cmd:
weed volume -max=0 -mserver="127.0.0.1:9333" -port=8080 -dir="/var/www/d1/sfs,/var/www/d2/sfs,/var/www/d3/sfs"
log:
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d1/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d2/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d3/sfs Permission: -rwxr-xr-x
F1006 15:09:26 17040 volume.go:142] 3 directories by -dir, but only 1 max is set by -max
What am I doing wrong? Thx.
 2020-10-06 09:05:30 -07:00
Chris Lu
36492c47ec adjust 2020-10-05 14:06:18 -07:00
Chris Lu
8f8e9ddbde adjust API 2020-10-05 09:47:07 -07:00
Konstantin Lebedev
e4f2d9eb4a We return etag using the same algorithm as aws s3 
https://teppen.io/2018/06/23/aws_s3_etags/
 2020-10-05 14:43:32 +05:00
Chris Lu
4fc673341f 2.03 2020-10-04 21:58:22 -07:00
Chris Lu
6b591b02af adjust log level 2020-10-04 16:21:43 -07:00
Устюжанин Антон Александрович
702b1cb876 fix: remove deleted peers if resumeState = true 2020-10-04 21:56:17 +05:00
Chris Lu
d522df0ecd adjust log level 2020-10-04 01:35:32 -07:00
Chris Lu
666859f7cd mount: read prefetching 2020-10-04 01:31:04 -07:00
Chris Lu
bc47835997 mount: avoid duplicated reads 2020-10-03 20:16:42 -07:00
Chris Lu
8d65ad1444 Revert "mount: adds read prefetching" 
This reverts commit 692f0614d1.
 2020-10-03 19:40:56 -07:00
Chris Lu
692f0614d1 mount: adds read prefetching 2020-10-03 16:50:09 -07:00
Chris Lu
707936f482 re-enable caching larger than 16MB 
revert 62ce85610e
 2020-10-03 14:12:38 -07:00
Chris Lu
2c20ef72ae fix starting problem 2020-10-03 13:56:38 -07:00
Chris Lu
d77e3c96e9 FUSE: use the mount directory permissions 2020-10-03 13:37:35 -07:00
Chris Lu
95ffed87c0
Merge pull request #1505 from ustuzhanin/remove_raft_state 
Resume raft state
 2020-10-03 10:38:47 -07:00
Устюжанин Антон Александрович
6fbfe73683 fix: resumeState 2020-10-03 22:16:47 +05:00
Устюжанин Антон Александрович
dc31b19469 fix: restore raft state 2020-10-03 14:03:41 +05:00
Chris Lu
f781cce500 s3: support object tagging 
* GetObjectTagging
* PutObjectTagging
* DeleteObjectTagging
 2020-10-02 22:21:51 -07:00
Устюжанин Антон Александрович
8c82fb7e5f fix: restore raft state 2020-10-02 23:01:20 +05:00
Chris Lu
9ab98fa912 s3 metrics adjust the label 2020-10-01 07:17:47 -07:00
Chris Lu
f1e879fe7a 2.02 2020-10-01 07:15:49 -07:00
Chris Lu
a1c01d716b volume: avoid deadlock when deleting volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1501
 2020-10-01 07:10:03 -07:00
Konstantin Lebedev
86329bbf2b label name is statusCode 2020-10-01 01:22:38 +05:00
Konstantin Lebedev
68463e92c1 add status code in S3RequestCounter 2020-10-01 00:59:39 +05:00
Chris Lu
79ab10e300 adjust help message 2020-09-30 09:32:00 -07:00
Chris Lu
55cb68c09c filer: replication follows master config if not specified 2020-09-30 09:15:55 -07:00
Chris Lu
b9887504e8 fix test 2020-09-27 23:19:50 -07:00
Chris Lu
f46eae284e adjust for test 2020-09-27 23:08:11 -07:00
Chris Lu
75aca5e13d 2.01 2020-09-27 23:02:41 -07:00
Chris Lu
e6552b5e1e filer: able to start s3 together 2020-09-27 23:00:43 -07:00
Chris Lu
474e2b6ac3 add a hard link marker to 16byte + maker, for future extensions 2020-09-27 22:38:30 -07:00
Chris Lu
c49e2bb9a3 adjust 2020-09-27 12:07:45 -07:00
Chris Lu
62ce85610e skip caching too large chunks 2020-09-27 11:58:48 -07:00
Chris Lu
9ad2dcca2b more tests 2020-09-27 11:42:51 -07:00
Chris Lu
e43d86c796 fix pre allocated volume size 2020-09-27 10:58:19 -07:00
Chris Lu
31fc7bb2e1 refactor 
adjust for faster test
 2020-09-27 10:41:29 -07:00
Chris Lu
a37535cd9f avoid non utf-8 in filename 
fix https://github.com/chrislusf/seaweedfs/issues/1493
 2020-09-26 00:13:39 -07:00
Chris Lu
41be7a4c8f filer: upload also set file size 
ensure works same as fuse mount
 2020-09-25 12:02:06 -07:00
Chris Lu
35f6518c36 weed upload: usePublicUrl was not being used on big files 
fix https://github.com/chrislusf/seaweedfs/issues/1492
 2020-09-25 02:41:38 -07:00
Chris Lu
22049dc2e6
Merge pull request #1491 from hilimd/master 
s3:  fix get object metadata problem
 2020-09-25 02:35:00 -07:00
limd
402aef8f30 s3: 
1.fix spark reading S3 directory wildcard problem
2.fix the problem of the spark history service writing S3 directory

reference git revsion number: b41b7ea4d0
 2020-09-25 14:37:02 +08:00
Chris Lu
8516517c48 filer store: Cassandra supports username/password 2020-09-24 21:31:06 -07:00
limd
ed7816681a Add cassandra authenticator mode 2020-09-25 11:22:47 +08:00
limd
370a98cf6b code style 2020-09-25 11:19:42 +08:00
limd
aee3fd08b9 code style 2020-09-25 11:04:32 +08:00
Chris Lu
043b063136
Merge pull request #1482 from hilimd/master 
Fix: s3 delete object
 2020-09-24 18:21:34 -07:00
limd
48c578410f mount: rollback default value 2020-09-25 09:18:52 +08:00
Chris Lu
dbf5327b98 s3: handle response-content-encoding 
fix https://github.com/chrislusf/seaweedfs/issues/1487
 2020-09-24 18:09:52 -07:00
Chris Lu
7726965a47 minor 2020-09-24 11:20:12 -07:00
Chris Lu
0790c6d605 fix empty hard link id 2020-09-24 11:16:43 -07:00
Chris Lu
1012df7bb5 switch hardlink id from int64 to bytes 2020-09-24 11:11:42 -07:00
Chris Lu
4856bce0ee adjust for metrics port 2020-09-24 10:21:23 -07:00
Chris Lu
d2d3aec3e1 consolidate to one metricsPort in "weed server" mode 2020-09-24 09:55:02 -07:00
Chris Lu
2e9099369e
Merge pull request #1489 from kmlebedev/promhttp 
Promhttp
 2020-09-24 09:50:48 -07:00
Chris Lu
2e7c361a0d hardlink deletion factors in hardlink counter 2020-09-24 09:43:52 -07:00
Chris Lu
1295347958 adjust hardlink update 
simplify logic, pass entity content directly to hard link. The "weed mount" handles the logic to calculate hard link counter.
 2020-09-24 09:43:00 -07:00
Konstantin Lebedev
98e9de6e11 fix style 2020-09-24 17:48:39 +05:00
Konstantin Lebedev
324e44d4b3 add start metrics server 2020-09-24 17:45:39 +05:00
limd
cbca14edc5 mount: fix k8s pvc and os mount directory permission bug 2020-09-24 18:07:16 +08:00
Chris Lu
5e239afdfc hardlink works now 2020-09-24 03:06:48 -07:00
limd
59e91e9c7e mount: fix k8s pvc mount directory permission 2020-09-24 10:25:45 +08:00
Chris Lu
c7d7b1a0f6
Merge pull request #1485 from LIBA-S/fix_oversized 
Correct the oversized state of volume after compaction
 2020-09-23 19:24:30 -07:00
Chris Lu
b80a2b3cc9
Merge pull request #1486 from LIBA-S/fix_race_condition 
Fix a race condition when handle VolumeLocationList
 2020-09-23 19:24:16 -07:00
LIBA-S
eecd6b5d35 Fix a race condition when handle VolumeLocationList 2020-09-23 20:56:51 +08:00
LIBA-S
0157798ebf Correct the oversized state of volume after compaction 2020-09-23 20:27:42 +08:00
limd
9b21ec27e4 mount: auto created dir set corrct umask fix bug 2020-09-23 18:35:37 +08:00
hilimd
0db149fb5f
Merge pull request #18 from chrislusf/master 
sync
 2020-09-23 17:32:05 +08:00
Chris Lu
f7a0ccb595 mount: auto created dir set corrct umask 2020-09-23 02:31:19 -07:00
Chris Lu
f0e325b6fc mount: auto created directory follow umask 2020-09-23 02:27:57 -07:00
limd
d506080c36 rollback 2020-09-23 14:29:53 +08:00
limd
8f9f29b773 fixed the problem of empty directory when S3 deleted the directory 2020-09-23 13:33:13 +08:00
limd
1892677b22 fixed the problem of empty directory when S3 deleted the directory 2020-09-23 13:15:06 +08:00
hilimd
85570f27a0
Merge pull request #17 from chrislusf/master 
sync
 2020-09-23 11:39:57 +08:00
limd
a99f63cb4d 1.add S3 copy directory function 
2.fixed the problem of empty directory when S3 deleted the directory
 2020-09-23 11:26:01 +08:00
Chris Lu
b61d33f251 expose only store wrapper to meta changes 2020-09-22 16:27:36 -07:00
Chris Lu
0adbb56cc1 rename 2020-09-22 16:24:13 -07:00
Chris Lu
3e52329cee Revert "Merge pull request #1479 from LIBA-S/fix_oversized" 
This reverts commit bd11f0b3e4, reversing
changes made to ec5b9f1e91.
 2020-09-22 15:05:37 -07:00