mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-23 00:27:55 +08:00
Code Issues Packages Projects Releases Wiki Activity
10,377 Commits 32 Branches 296 Tags 152 MiB
f08f95ac80
Commit Graph

228 Commits

Author SHA1 Message Date
Chris Lu
cefe66f159 dedicated upload processor 
avoid thundering effect of overloading volume servers
 2021-04-01 02:21:40 -07:00
Chris Lu
1f984d2645 refactor buffer pool 2021-04-01 02:20:00 -07:00
Chris Lu
3b2a9c98ef fix nil 2021-03-07 11:45:35 -08:00
Chris Lu
df72dc206d stats collect chunk upload retry count 2021-03-07 11:26:15 -08:00
Chris Lu
44bdfb2d15 filer: avoid encryption and compression at the same time 
fix https://github.com/chrislusf/seaweedfs/issues/1828
 2021-02-22 12:22:49 -08:00
bingoohuang
352ac2f271 Merge remote-tracking branch 'origin/master' 2021-02-18 14:05:51 +08:00
bingoohuang
eab6e31d34 use backticks instead of double quotes to avoid escaped additionally in regex 2021-02-18 14:05:28 +08:00
Chris Lu
3575d41009 go fmt 2021-02-17 20:57:08 -08:00
Chris Lu
6daa932f5c refactoring to get master function, instead of passing master values directly 
this will enable retrying later
 2021-02-17 20:55:55 -08:00
Chris Lu
dd9f3a0104 add sleep between upload retries 2021-02-17 20:55:11 -08:00
Chris Lu
487e435679 adjust http max idle connections per host 
related to https://github.com/chrislusf/seaweedfs/issues/1802
 2021-02-12 03:47:15 -08:00
Chris Lu
821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
Chris Lu
00707ec00f mount: outsideContainerClusterMode proxy through filer 
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
 2021-01-24 19:01:58 -08:00
Chris Lu
4b5adc60b4 upload result changes to the right output variable 2021-01-03 01:44:22 -08:00
Chris Lu
3433accb1b weed upload: add TTL 2020-12-30 15:46:07 -08:00
Chris Lu
1bf22c0b5b go fmt 2020-12-16 09:14:05 -08:00
Chris Lu
0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu
e9cd798bd3 adding volume type 2020-12-13 00:58:58 -08:00
Chris Lu
0da7ecfd29 go fmt 2020-11-26 11:22:30 -08:00
Chris Lu
92f906b6fc remove zstd 
fix https://github.com/chrislusf/seaweedfs/issues/1629
 2020-11-21 13:06:45 -08:00
Chris Lu
dc304342b2 fs.configure: configurable volume growth 2020-11-17 01:00:02 -08:00
Chris Lu
b04375a3d1 use alternative assign request 2020-11-17 00:36:21 -08:00
Chris Lu
0ea5c087ce go fmt 2020-11-15 16:59:28 -08:00
Chris Lu
95c0de285d refactoring 2020-11-15 16:58:48 -08:00
Chris Lu
a29abd8600 add a {X-,}Idempotency-Key header for http POST 
fix https://github.com/chrislusf/seaweedfs/issues/1592

according to https://github.com/golang/go/issues/19943
 2020-11-11 12:56:53 -08:00
Chris Lu
8708e7a64d print out data size 2020-11-11 12:51:44 -08:00
Chris Lu
0ecff9521d avoid []byte reuse 2020-11-02 23:27:54 -08:00
Chris Lu
031aa70da6 Revert "revert bytebufferpool, seems problematic" 
This reverts commit e0fcab47c0.
 2020-11-02 19:11:25 -08:00
Chris Lu
e0fcab47c0 revert bytebufferpool, seems problematic 2020-11-02 13:49:38 -08:00
Chris Lu
c057dd1f64 add back upload retries 2020-10-30 02:16:34 -07:00
Chris Lu
19772d70d7 print for debugging 2020-10-24 00:12:02 -07:00
Chris Lu
ef9c32ea0d adjust compression threshold 2020-10-23 23:05:55 -07:00
Chris Lu
29c9fa2ef2 byte buffer for uploading 2020-10-23 11:29:51 -07:00
Chris Lu
2579edbc60 fix bug found by tests 2020-10-23 10:05:40 -07:00
Chris Lu
e0c8507d93 memory pool to reduce memory allocation 2020-10-23 00:41:40 -07:00
Chris Lu
c31b254248 mount: shortcut when there is only one chunk 2020-10-20 23:48:29 -07:00
Chris Lu
f64252023e Revert "a better byte buffer pool" 
This reverts commit 59c6422777.
 2020-10-20 23:21:54 -07:00
Chris Lu
59c6422777 a better byte buffer pool 2020-10-20 22:53:58 -07:00
Chris Lu
35f6518c36 weed upload: usePublicUrl was not being used on big files 
fix https://github.com/chrislusf/seaweedfs/issues/1492
 2020-09-25 02:41:38 -07:00
Chris Lu
daf0a449f7 properly cancel context for streaming grpc 2020-09-09 12:07:15 -07:00
Chris Lu
387ab6796f filer: cross cluster synchronization 2020-09-09 11:21:23 -07:00
Chris Lu
50f7e2e89f add log messages 2020-08-26 09:51:20 -07:00
Chris Lu
5f55a87101 close http response 2020-08-24 00:32:44 -07:00
Chris Lu
0b301b0b47 consume all response body in order to release requests 2020-08-23 21:32:29 -07:00
Chris Lu
003d48da21 adjust logs 2020-08-15 19:55:28 -07:00
Chris Lu
83cad3da79 add retry file upload 2020-08-11 20:30:11 -07:00
Chris Lu
20e2ac1add filer: store md5 metadata for files uploaded by filer 
fix https://github.com/chrislusf/seaweedfs/issues/1412
 2020-08-06 10:04:17 -07:00
Chris Lu
4ecfa9879d volume: report Content-MD5 in response header 2020-08-06 05:22:53 -07:00
Chris Lu
1b3a80dd3d non-fatal error 2020-08-01 13:46:52 -07:00
Chris Lu
bbc8ae110f report errors 2020-07-14 11:27:00 -07:00
Chris Lu
ddec7b2bb9 go fmt 2020-07-14 07:34:16 -07:00
Chris Lu
e4ef3814e7 fix compilation 2020-07-10 10:02:59 -07:00
Chris Lu
2ff37ccdbd testing parsing the upload 
The mime type is always the value passed in.

Compress or not depends on the content detection, file name extension, and compression ratio.

If the content is already compressed, need to know the content size.
 2020-06-24 11:39:12 -07:00
Chris Lu
fe60db404a prepare for zstd 2020-06-23 09:12:02 -07:00
Chris Lu
3080c197e3 rename UnCompressData to DecompressData 2020-06-20 08:16:16 -07:00
Chris Lu
97239ce6f1 rename filechunk is_gzipped to is_compressed 2020-06-20 08:15:49 -07:00
Chris Lu
e912fd15e3 renaming 2020-06-19 22:45:27 -07:00
Chris Lu
16fe132a20 detect mime type before replicating to other volume servers 2020-06-19 22:11:36 -07:00
Chris Lu
39e72fb23c go fmt 2020-05-10 03:50:30 -07:00
Chris Lu
78422c2f69
Merge pull request #1298 from song-zhang/master 
add fsync option for benchmark
 2020-04-30 20:16:42 -07:00
Chris Lu
871efa4fc1 refactoring 
some previous chunk etag was using md5, which should be wrong.
 2020-04-30 17:20:47 -07:00
zhangsong
d819865fe7 add fsync option for benchmark 2020-04-30 23:31:08 +08:00
Chris Lu
9cacaf5eca better detect file mime type 2020-04-14 11:32:31 -07:00
Chris Lu
3f1d79512f only detect when not sure about the types 2020-04-14 11:02:05 -07:00
Chris Lu
d2d1597850 volume: detect gzip effectiveness only when content type is empty 
revert part of 2286eda575
 2020-04-14 10:52:53 -07:00
Chris Lu
8a73276602 refactor a bit 2020-04-08 22:33:36 -07:00
Chris Lu
bf270d9e8c filer: able to tail meta data changes 2020-04-05 00:51:16 -07:00
Chris Lu
eedd33dda3 refactoring 2020-03-28 13:41:58 -07:00
Chris Lu
e1911760a7 refactoring 2020-03-27 00:30:55 -07:00
Chris Lu
2286eda575 sampling whether the data can be gzipped 2020-03-26 23:50:48 -07:00
Chris Lu
a5ce359999 refactoring 2020-03-26 22:46:49 -07:00
Chris Lu
6d87a0c5f5 batch delete orphan data by file keys 2020-03-25 02:21:15 -07:00
Chris Lu
6ff9e2835e filer: skip resizing image if width or height larger than original image 
fix https://github.com/chrislusf/seaweedfs/issues/1239#issuecomment-602140779
 2020-03-21 22:16:00 -07:00
Chris Lu
308688c8d0 filer: add back image resizing capability 2020-03-20 20:31:11 -07:00
Chris Lu
f251d03673 refactoring 2020-03-20 15:46:16 -07:00
Chris Lu
11372dd548 refactoring 2020-03-20 15:38:04 -07:00
Chris Lu
d848d08944 use public url also for deletions 2020-03-18 11:16:45 -07:00
Chris Lu
6566c8e114 weed upload: add usePublicUrl option 
fix https://github.com/chrislusf/seaweedfs/issues/1236
 2020-03-18 10:50:53 -07:00
Chris Lu
f9b3750ad1 volume: handle repeated writes response 2020-03-15 04:16:00 -07:00
Chris Lu
ee635dcc36 refactoring 2020-03-14 14:08:00 -07:00
Chris Lu
18cda6adbb cleaner fix 2020-03-14 13:55:32 -07:00
Chris Lu
17efa361f9 fix possible nil 2020-03-14 13:53:03 -07:00
Chris Lu
3934c69757 clean up 2020-03-08 21:54:47 -07:00
Chris Lu
2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu
5ac6297c68 adjust parameter names 2020-03-08 17:03:37 -07:00
Chris Lu
9228ff192c encryption works 2020-03-07 11:08:57 -08:00
Chris Lu
0375ce2c2e filer: set mime type from volume server 2020-03-07 07:25:15 -08:00
Chris Lu
ea1169dc80 filer cipher: single chunk http POST and PUT and read 2020-03-07 06:08:08 -08:00
Chris Lu
13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu
f90c43635d refactoring 2020-03-04 00:39:47 -08:00
Chris Lu
892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu
6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu
33b3bd467c Revert "HEAD operation changes to fasthttp" 
This reverts commit 58f126fd27.
 2020-02-10 13:43:53 -08:00
Chris Lu
58f126fd27 HEAD operation changes to fasthttp 2020-02-10 09:13:29 -08:00
Chris Lu
72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu
8dd260623e protect against nil deletion results 
fix https://github.com/chrislusf/seaweedfs/issues/1180
 2020-01-08 21:50:59 -08:00
Chris Lu
392678f8f3 upload skipping mimetype if not needed 2019-12-15 20:57:08 -08:00
Chris Lu
d0b423bbc0 filer: increase directory listing pagination size 2019-12-12 09:11:10 -08:00
Lei Liu
46755ea1e1 fix master maintenance logic 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-11-12 14:46:10 +08:00
zhangsong
61fa485700 add volume number param in assign operation 2019-11-10 20:11:03 +08:00