mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-02 16:39:16 +08:00
Code Issues Packages Projects Releases Wiki Activity
4,813 Commits 32 Branches 296 Tags 149 MiB
b5e2be635a
Commit Graph

837 Commits

Author SHA1 Message Date
Chris Lu
1dd2c76532 refactoring 2020-07-12 17:31:24 -07:00
Chris Lu
e7c04af1d0 skip http.StatusPartialContent in case of error and superfluous response.WriteHeader 2020-07-10 08:56:23 -07:00
李远军
6608cb5f43 Revert "remove fixJpgOrientation" 
This reverts commit de5ca9b2
 2020-07-10 10:08:36 +08:00
Chris Lu
a2eb680f34 filer: default filer store directory 
1. set default filer store directory
2. set peers, avoiding empty string counted as 1.
 2020-07-07 23:06:48 -07:00
Chris Lu
3c269da37f fix compilation 2020-07-06 09:46:24 -07:00
Chris Lu
d198e8c6d6 filer: support cross filer meta data sync if sharing the same store 2020-07-06 09:24:54 -07:00
Chris Lu
0b8cc9b626 aggregate multiple filer metadata chagne events 2020-07-05 23:05:02 -07:00
Chris Lu
49929e0869 rename 2020-07-05 15:52:36 -07:00
Chris Lu
70d8a3a1d3 add SubscribeLocalMetadata without checking persisted meta logs 2020-07-05 15:50:07 -07:00
Chris Lu
55e40b08fc refactoring 2020-07-05 15:43:06 -07:00
Chris Lu
881e0fde2e rename 2020-07-05 08:55:09 -07:00
Chris Lu
4d1484628a refactoring 2020-07-03 16:41:30 -07:00
Chris Lu
31e23e9783 filer: support active<=>active filer replication 2020-06-30 22:53:57 -07:00
Chris Lu
df75b5c98d passing ctx for metadata logging 2020-06-28 14:34:51 -07:00
Chris Lu
a4dc25cd22 volume: generate ec shard first, before generating the ec index file. 
possible fix https://github.com/chrislusf/seaweedfs/issues/1364

the ec index file is used to determine whether the ec files are ready or not
 2020-06-25 11:06:58 -07:00
Chris Lu
3b638d3994 add more ec encoding logging 2020-06-25 09:43:38 -07:00
Chris Lu
fe60db404a prepare for zstd 2020-06-23 09:12:02 -07:00
Chris Lu
a4e3cffe0a S3: DeleteObject API also support deleting a folder recursively 2020-06-22 10:01:00 -07:00
Chris Lu
de5ca9b258 remove fixJpgOrientation 2020-06-20 12:50:40 -07:00
Chris Lu
3080c197e3 rename UnCompressData to DecompressData 2020-06-20 08:16:16 -07:00
Chris Lu
ca3516ac6d adjust protoc 2020-06-20 08:00:25 -07:00
Chris Lu
e912fd15e3 renaming 2020-06-19 22:45:27 -07:00
Chris Lu
16fe132a20 detect mime type before replicating to other volume servers 2020-06-19 22:11:36 -07:00
Chris Lu
044841c885 master: always clear previous master meta data directory 2020-06-19 20:42:16 -07:00
Chris Lu
ae1994cbc1 erasure coding: fix cases where there are no .ecj files 2020-06-18 09:52:35 -07:00
Chris Lu
c66f357715 report error correctly 2020-06-10 10:59:25 -07:00
Chris Lu
c38066be89 also add it for webdav 2020-06-09 18:07:03 -07:00
Evgenii Kozlov
2410ffbd79 Name fix 2020-06-05 19:00:38 +03:00
Evgenii Kozlov
0e0db70f55 Set volumes ReadOnly if low free disk space 2020-06-05 18:18:15 +03:00
Chris Lu
393ab07c7c disable pulse seconds 
there are a few other hard coded timing settings, which will cause disconnected volume servers.
 2020-06-04 10:52:01 -07:00
Chris Lu
bc2ec6774d inject git version into build 2020-06-02 00:10:38 -07:00
bingoohuang
1a642b9876 add Volume Ids column only for max 100 volumes for convenience in the master ui. 2020-05-29 15:37:58 +08:00
bingoohuang
accb4964b7 util.BytesToHumanReadable 2020-05-29 10:00:07 +08:00
bingoohuang
21d0a013d5 rename 'needle' which collides with imported package name 2020-05-28 17:00:00 +08:00
bingoohuang
41cab5d69f log JSON response if httpStatus >= 400 2020-05-28 08:56:55 +08:00
Chris Lu
d4235afe4d support multiple locks 2020-05-25 18:39:53 -07:00
bingoohuang
ecdeef8c66 simplify func(w http.ResponseWriter, r *http.Request) to http.HandlerFunc 2020-05-25 21:00:12 +08:00
Chris Lu
91b91d6cb7 add error to avoid copying not found volume 
fix https://github.com/chrislusf/seaweedfs/issues/1317
 2020-05-17 20:20:12 -07:00
Chris Lu
3fbf635308 Merge branch 'master' into msg_channel 2020-05-13 03:54:15 -07:00
Chris Lu
3a5e4769cf prevent empty locations returned 
fix https://github.com/chrislusf/seaweedfs/issues/1313
 2020-05-13 03:46:38 -07:00
Chris Lu
dfccc3c263 able to read chan and write chan 2020-05-08 02:47:22 -07:00
Chris Lu
1e3e4b3072 add broker connects to filer 2020-05-05 02:05:28 -07:00
Chris Lu
871efa4fc1 refactoring 
some previous chunk etag was using md5, which should be wrong.
 2020-04-30 17:20:47 -07:00
Chris Lu
9e72e9e4b8 able to subscribe any topic from any point of time 2020-04-29 17:40:08 -07:00
Chris Lu
ed3cf811f5 refactoring 2020-04-29 13:26:02 -07:00
Chris Lu
8d42a1263b meta subscription: update the last read time 2020-04-28 17:30:04 -07:00
Chris Lu
31583b5d6c master and volume server: avoid race condition 
The volume server may disconnect and reconnect to the same master.

The master's unregistration may happen after the reconnection. Thus the volume server will disappear.
 2020-04-28 17:29:10 -07:00
Chris Lu
98edec7c3e remove printouts 2020-04-28 16:45:48 -07:00
Chris Lu
fb81f12686 filer: simplify image resize 2020-04-28 00:05:57 -07:00
Chris Lu
5c57297bd1 metadata log: read from any timestamp 2020-04-28 00:05:57 -07:00
wuyuxiang
6850d28d6b refacotr: move signal handling and pprof to grace package 2020-04-28 14:10:23 +08:00
Chris Lu
b52b8ec685 Hadoop: fix entry not found for HCFS 
also fix cipher related changes.
 2020-04-26 05:21:54 -07:00
Chris Lu
662b5d0cf7 support semicolon seperated command lines 2020-04-23 14:01:46 -07:00
Chris Lu
73564e6a01 master: add cluster wide lock/unlock operation in weed shell 
fix https://github.com/chrislusf/seaweedfs/issues/1286
 2020-04-23 13:37:31 -07:00
Chris Lu
bdc337a719 make lock/unlock optional 2020-04-23 03:50:05 -07:00
Chris Lu
842e3301ee make second lock instant 2020-04-23 03:32:18 -07:00
Chris Lu
30ee4f3291 add exclusive lock library on shell 2020-04-23 02:31:04 -07:00
Chris Lu
77873b832b add master side code for cluster wise exclusive lock 2020-04-23 01:55:44 -07:00
Chris Lu
69eda558ef reduce the connections 
fix https://github.com/chrislusf/seaweedfs/issues/1277

this avoids sending error to wdclients, and clients retry the connections
 2020-04-22 20:43:34 -07:00
Chris Lu
e24b25de78 async meta caching: can stream updates now 2020-04-21 21:16:13 -07:00
Chris Lu
27128c7875 adjust logging 2020-04-21 16:13:13 -07:00
Chris Lu
cb3985be70 go fmt 2020-04-20 17:48:06 -07:00
Chris Lu
ebfab42a50 refactoring 2020-04-20 00:08:47 -07:00
Chris Lu
2955b96ef1 refactor 2020-04-19 23:54:32 -07:00
Chris Lu
f373232227
Merge pull request #1280 from bukton123/filer_mongodb 
Add new filer support mongodb
 2020-04-19 20:35:19 -07:00
Chris Lu
076c8bd3bc filer master start up with default ip address instead of just localhost 2020-04-18 15:17:27 -07:00
bukton
290c6b7f01 Merge remote-tracking branch 'origin/master' into filer_mongodb 
# Conflicts:
#	go.mod
#	go.sum
#	weed/server/filer_server.go
 2020-04-19 00:21:45 +07:00
bukton
cd9cccec55 add import mongodb to filer server 2020-04-18 22:30:13 +07:00
Chris Lu
826f96b6f3 fix unreachable code 2020-04-17 10:04:28 -07:00
Chris Lu
bda82f61bc filer: able to append to a file 2020-04-17 02:28:09 -07:00
Chris Lu
f5a748d33c refactoring 2020-04-16 02:55:09 -07:00
Chris Lu
ce4b369be2 scaffolding messaging 2020-04-16 02:21:23 -07:00
Reed
aada5b9aa7 making volume stats size info readable. 2020-04-16 10:08:08 +08:00
Chris Lu
2b5c4fbbf3 tiered caching 
1/4 for small less than 1MB files. 1/4 for 1~4MB files, 1/2 for bigger than 4MB files
 2020-04-13 22:19:27 -07:00
Chris Lu
7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu
8e23dc078b refactoring 2020-04-12 20:48:21 -07:00
Chris Lu
5361f999ed shell: only one shell is allowed to connect to the cluster 
fix https://github.com/chrislusf/seaweedfs/issues/1266
 2020-04-12 17:48:39 -07:00
Chris Lu
d30483d642 re-enable system logs 2020-04-12 14:03:07 -07:00
Chris Lu
6f948e4887 remove configurable topics folder location 2020-04-12 13:07:59 -07:00
Chris Lu
b5a713be68 filer store: add redis2 redis_cluster2 as default redis store 2020-04-12 02:50:41 -07:00
Chris Lu
1c65656fb4 s3: add option to fsync buckets 2020-04-11 23:37:10 -07:00
Chris Lu
e4af63a721 volume server: accept fsync=true in write requests 2020-04-11 21:39:16 -07:00
Chris Lu
df97da25f9 mount: add on disk caching 2020-04-11 21:12:41 -07:00
Chris Lu
d7f3acb2c0 refactor 2020-04-11 12:45:24 -07:00
Chris Lu
e909b55633 refactoring 2020-04-11 12:39:46 -07:00
Chris Lu
1101a42e5c mv : create new folder, move children, and delete old folder 2020-04-10 01:37:03 -07:00
Chris Lu
f6a7e79dc3 weed shell: simplify CLI option for filer 2020-04-08 23:57:15 -07:00
Chris Lu
dc08e4098f add etag only for PUT or large chunked uploads 2020-04-08 09:13:26 -07:00
Chris Lu
ec2eb8bc48 add If-None-Match and If-Modified-Since 
fix https://github.com/chrislusf/seaweedfs/issues/1269
 2020-04-08 08:12:00 -07:00
Chris Lu
745f5d2a25 fix setting default value 2020-04-07 01:58:48 -07:00
Chris Lu
54debdc6f7 filer: fix configuration settings 2020-04-07 01:30:53 -07:00
Chris Lu
f14b6a09a8 remove duplicated update events 2020-04-05 13:22:15 -07:00
Chris Lu
91da7057b1 refactoring 2020-04-05 13:11:43 -07:00
Chris Lu
2d43f85577 watch entries with common path prefix 2020-04-05 12:51:21 -07:00
Chris Lu
7bc3c93512 add util.PathJoin 2020-04-05 12:40:46 -07:00
Chris Lu
bf270d9e8c filer: able to tail meta data changes 2020-04-05 00:51:16 -07:00
Chris Lu
6e776aaafc loop through all masters 2020-04-03 00:47:48 -07:00
Chris Lu
4a5e381feb fail fast if filer configured wrong masters 
fix https://github.com/chrislusf/seaweedfs/issues/1257
 2020-04-03 00:47:33 -07:00
Chris Lu
aaf5e658cc adjust parameter name 2020-04-03 00:41:05 -07:00
Chris Lu
6c96e51414 re-ordering 2020-04-03 00:40:54 -07:00
Chris Lu
fee3319cbf handle gzip for image resizing 2020-04-02 23:44:24 -07:00
Chris Lu
73bc286377 volume: passing ext info via URL 2020-04-02 21:23:15 -07:00
Chris Lu
cea52a4faf volume copying adds cleaning up on error 
fix https://github.com/chrislusf/seaweedfs/issues/1253
 2020-04-02 15:36:55 -07:00
Chris Lu
c446438ca5
Merge pull request #1255 from levenlabs/ignore 
Added treat_replication_as_minimums master toml option
 2020-04-01 16:48:58 -07:00
James Hartig
eae3f27c80 Added treat_replication_as_minimums master toml option 2020-04-01 19:08:48 -04:00
Chris Lu
50a5018b7f writing meta logs is working 2020-03-30 01:19:33 -07:00
Chris Lu
9dc0b1df8f refactoring to compile for windows OS 
windows os does not like to work with fuse
 2020-03-29 21:07:55 -07:00
Chris Lu
a75d50bbb8 FUSE: add configurable in memory chunk cache size 2020-03-28 14:07:16 -07:00
Chris Lu
826bc0b7e3 FUSE: add chunk cache for recently accessed file chunks 2020-03-28 13:43:31 -07:00
Chris Lu
eedd33dda3 refactoring 2020-03-28 13:41:58 -07:00
Chris Lu
d1439c5bd3 fix FUSE read for large files 
FUSE expects ReadAt do not return partial filled buffer with a nil error.
 2020-03-27 04:50:51 -07:00
Chris Lu
34a0d424f8 FUSE mount: report max volume count available disk space 2020-03-26 23:06:21 -07:00
Chris Lu
a5ce359999 refactoring 2020-03-26 22:46:49 -07:00
Chris Lu
6d87a0c5f5 batch delete orphan data by file keys 2020-03-25 02:21:15 -07:00
Chris Lu
cddcb5cafe same booleans as before 2020-03-23 01:25:23 -07:00
Chris Lu
f84c288852 purge 2020-03-23 01:14:55 -07:00
Chris Lu
a5aa8be19b Update webdav_server.go 2020-03-23 01:14:42 -07:00
Chris Lu
c0f0fdb3ba refactoring 2020-03-23 00:01:34 -07:00
Chris Lu
c16dc57a58 fix max volume count reporting 2020-03-22 18:33:19 -07:00
Chris Lu
3137777d83 volume: automatically detect max volume count 2020-03-22 16:21:42 -07:00
Chris Lu
9f2c9b666b adjust log level 2020-03-22 01:39:08 -07:00
Chris Lu
ae2ee379c0 consistent 64bit size 2020-03-22 01:37:46 -07:00
Chris Lu
e93e986727 webdav: stream read 2020-03-22 01:30:27 -07:00
Chris Lu
65d2ea9fb0 FUSE mount: stream read data with buffer 
fix https://github.com/chrislusf/seaweedfs/issues/1244
 2020-03-22 01:00:36 -07:00
Chris Lu
308688c8d0 filer: add back image resizing capability 2020-03-20 20:31:11 -07:00
Chris Lu
f251d03673 refactoring 2020-03-20 15:46:16 -07:00
Chris Lu
11372dd548 refactoring 2020-03-20 15:38:04 -07:00
Chris Lu
c4bea45099 S3 API: fix DeleteMultipleObjectsHandler 
fix https://github.com/chrislusf/seaweedfs/issues/1241
 2020-03-20 14:17:31 -07:00
Chris Lu
d848d08944 use public url also for deletions 2020-03-18 11:16:45 -07:00
Chris Lu
bec6ec7db6 go fmt 2020-03-17 10:01:55 -07:00
Chris Lu
c3cb6fa1d7 volume: compaction can cause readonly volumes 
address https://github.com/chrislusf/seaweedfs/issues/1233
 2020-03-17 09:43:57 -07:00
Chris Lu
560df51def refactoring 2020-03-15 03:11:26 -07:00
Chris Lu
7b37178716 filer: close stores if interrupted 2020-03-14 20:30:26 -07:00
Chris Lu
e85da50ad4 remove tikv, move its support to "tikv" branch 2020-03-11 20:59:00 -07:00
Chris Lu
4237a813cc skip printouts 2020-03-11 01:13:54 -07:00
Chris Lu
d214cefc2e filer: list diretory pagination make up for the expired entries 2020-03-09 23:28:01 -07:00
Chris Lu
0871d2cff0 volume: fix memory leak during compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1222
 2020-03-09 22:29:02 -07:00
Chris Lu
89eb05b50f filer: support TTL for all filer stores 2020-03-09 01:02:01 -07:00
Chris Lu
8a899992f2 filer: fix ttl parsing 
fix https://github.com/chrislusf/seaweedfs/issues/1225
 2020-03-09 00:16:10 -07:00
Chris Lu
2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu
e04c1a1361 filer: remove chunk manifest file support 
This is not needed for filer
 2020-03-08 17:03:27 -07:00
Chris Lu
9b3109a5d8 filer: processing all response headers, no pass through to volume server 
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
 2020-03-08 15:42:44 -07:00
Chris Lu
11fceaf2f7 refactoring 2020-03-07 18:09:30 -08:00
Chris Lu
1a75fbfce0 refactoring 2020-03-07 18:06:48 -08:00
Chris Lu
afb20de14c breaks dependency loop 2020-03-07 17:01:39 -08:00
Chris Lu
9228ff192c encryption works 2020-03-07 11:08:57 -08:00
Chris Lu
0375ce2c2e filer: set mime type from volume server 2020-03-07 07:25:15 -08:00
Chris Lu
dba35404e4 filer: HEAD response add content type 2020-03-07 06:08:08 -08:00
Chris Lu
ea1169dc80 filer cipher: single chunk http POST and PUT and read 2020-03-07 06:08:08 -08:00
Chris Lu
13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu
f90c43635d refactoring 2020-03-04 00:39:47 -08:00
Chris Lu
9b7dae8037 filer: fix uploaded chunked file size reporting bug. 2020-03-02 20:27:14 -08:00
Chris Lu
dc40413847 fix error 2020-03-01 22:21:13 -08:00
Chris Lu
6a8484b4ae master able to list all master clients by type 2020-03-01 22:13:47 -08:00
Chris Lu
555413d9fc weed queue starts 2020-02-27 00:07:13 -08:00
Chris Lu
8c1da71402 remove dead code 2020-02-26 17:27:49 -08:00
Chris Lu
0156e2975a mount: add mode to run external to SeaweedFS container cluster 2020-02-26 16:46:01 -08:00
Chris Lu
97ab8a1976 remove ctx if possible 2020-02-25 22:23:59 -08:00
Chris Lu
892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu
0841bedb15 move filer assign volume grpc errror to response 2020-02-25 17:15:09 -08:00
Chris Lu
bc38b72a20 s3: implemented DeleteMultipleObjects 2020-02-25 14:38:36 -08:00
Chris Lu
6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu
5bcb44eda9 filer: move buckets folder to filer.toml since it should not be changed often 2020-02-24 14:42:57 -08:00
Chris Lu
d8dec2323b s3: move buckets folder configuration to filer 2020-02-24 14:34:14 -08:00
Chris Lu
b06b7ca6e6 adjust UI 2020-02-23 22:01:03 -08:00
Yoni Nakache
0d60bb4427
display bug, header is Free but data was Used 2020-02-24 07:47:27 +02:00
Chris Lu
346a1cf0b9
Merge pull request #1207 from LazyDBA247-Anyvision/update-stats-and-status-volume-server 
Update stats and status volume server
 2020-02-23 18:05:25 -08:00
LazyDBA247-Anyvision
4ff513d64d staus route: add DiskStatuses for disk in the volume server status 
whem monitoring server, better to know the status of the disks & volumes in a single route.
 2020-02-23 23:27:09 +02:00
Yoni Nakache
cd4c9a365b
DiskStats: adding Total & Percent Usage 
making relevant data visible and readable.
 2020-02-23 22:33:47 +02:00
Chris Lu
f55f49970f move volume server stats to grpc 2020-02-21 21:45:03 -08:00
Chris Lu
9b6296e77a volume: add grpc file read operation 
This is added more for performance benchmarking.
 2020-02-14 00:37:32 -08:00
Chris Lu
2a6db0fd43 volume: return 204 for unchanged file uploads 
fix https://github.com/chrislusf/seaweedfs/issues/1196
 2020-02-11 09:45:02 -08:00
Chris Lu
382ff218d3 filer: set file size, streaming chunk file uploading 
fix https://github.com/chrislusf/seaweedfs/issues/1193
 2020-02-03 17:04:06 -08:00
Chris Lu
40ae533fa3 shell: add volume.configure.replication to change replication for a volume 
fix https://github.com/chrislusf/seaweedfs/issues/1192
 2020-02-02 15:37:23 -08:00
Chris Lu
d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
Chris Lu
72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu
c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu
d4cde5df49 return empty response if not found 2020-01-24 18:07:34 -08:00
Chris Lu
3cdb27fafc adjust logging 2020-01-24 17:55:39 -08:00
Chris Lu
e6f9f5da3a adjust logging 2020-01-24 17:26:03 -08:00
Chris Lu
d4e75a0d18 filer: option to create file only if it is new, O_EXCL 2020-01-22 11:42:40 -08:00
Chris Lu
9b01a99d9a adjust logging 2020-01-21 22:45:50 -08:00
Chris Lu
1b0bfbaf59 refactoring 2020-01-19 23:59:46 -08:00
Chris Lu
4e731f1c8b volume: copy volumes also include .vif file 2020-01-04 11:28:29 -08:00
Chris Lu
df636e4c14 remove .vif files 2020-01-03 12:46:42 -08:00
Chris Lu
3eafec4b29 volume: add option to limit file size 2020-01-03 00:37:26 -08:00
Chris Lu
86d4b18eb5 filer: optional recursive deletion 
fix https://github.com/chrislusf/seaweedfs/issues/1176
 2019-12-31 11:52:54 -08:00
Chris Lu
509f314350 keep .vif file for ec to normal volume conversion 2019-12-28 21:52:55 -08:00
Chris Lu
0cc8347a3a go fmt 2019-12-28 21:13:10 -08:00
Chris Lu
8fbc0a9163 fix edge cases 2019-12-28 12:59:31 -08:00
Chris Lu
37b64a50b4 ec: generate and copy .vif file 2019-12-28 12:44:59 -08:00
Chris Lu
2000284435 rename volume tier info to volume info 2019-12-28 11:21:49 -08:00
Chris Lu
b3c83f9404 fix spelling 2019-12-28 11:16:10 -08:00
Chris Lu
48d28d3eb2 tier: support remote file attributes and remember the file extension 2019-12-25 21:37:24 -08:00
Chris Lu
d960b3474a tier storage: support downloading the remote dat files 2019-12-25 09:53:13 -08:00
Chris Lu
7ba6be2312 volume ui add readonly status 2019-12-24 14:55:26 -08:00
stlpmo
f87ccbddb4 remove the http 304 response body 2019-12-24 17:56:50 +08:00
Chris Lu
8a2d529759 go fmt 2019-12-23 12:48:53 -08:00
Chris Lu
09ca936c78 shell: add ec.decode command 2019-12-23 12:48:20 -08:00
Chris Lu
36ddca9d1f filer: return not found 404 status if not found 
fix https://github.com/chrislusf/seaweedfs/issues/1160
 2019-12-18 21:04:40 -08:00