mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-18 12:47:50 +08:00
Code Issues Packages Projects Releases Wiki Activity
3,558 Commits 32 Branches 296 Tags 152 MiB
ed9e4dc749
Commit Graph

2096 Commits

Author SHA1 Message Date
Chris Lu
2a45897237 broker: read cipher value from filer 2020-04-17 02:29:00 -07:00
Chris Lu
bda82f61bc filer: able to append to a file 2020-04-17 02:28:09 -07:00
Chris Lu
722b14337e add cipher option to meta data updates 2020-04-17 01:26:27 -07:00
Chris Lu
7d3672c60a simplify 2020-04-17 00:00:48 -07:00
Chris Lu
508f3490a0 update messaging proto 2020-04-16 03:29:57 -07:00
Chris Lu
f5a748d33c refactoring 2020-04-16 02:55:09 -07:00
Chris Lu
ce4b369be2 scaffolding messaging 2020-04-16 02:21:23 -07:00
Reed
aada5b9aa7 making volume stats size info readable. 2020-04-16 10:08:08 +08:00
Chris Lu
9cacaf5eca better detect file mime type 2020-04-14 11:32:31 -07:00
Chris Lu
3f1d79512f only detect when not sure about the types 2020-04-14 11:02:05 -07:00
Chris Lu
d2d1597850 volume: detect gzip effectiveness only when content type is empty 
revert part of 2286eda575
 2020-04-14 10:52:53 -07:00
Chris Lu
23715af402
Merge pull request #1273 from wyxloading/config_race 
Fix data race about config
 2020-04-13 22:24:51 -07:00
Chris Lu
2b5c4fbbf3 tiered caching 
1/4 for small less than 1MB files. 1/4 for 1~4MB files, 1/2 for bigger than 4MB files
 2020-04-13 22:19:27 -07:00
Chris Lu
f282ed444b refactoring 2020-04-13 21:58:10 -07:00
wuyuxiang
b556d3d035 Fix data race about config 2020-04-14 10:22:40 +08:00
Chris Lu
d8f5985e5e add logs 2020-04-13 12:58:45 -07:00
Chris Lu
5c9e91393f 1.74 2020-04-13 10:25:48 -07:00
Chris Lu
7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu
8e23dc078b refactoring 2020-04-12 20:48:21 -07:00
Chris Lu
b062393f3f refactoring 2020-04-12 17:51:31 -07:00
Chris Lu
5361f999ed shell: only one shell is allowed to connect to the cluster 
fix https://github.com/chrislusf/seaweedfs/issues/1266
 2020-04-12 17:48:39 -07:00
Chris Lu
eb16bb0287 go mod cleanup 2020-04-12 16:44:58 -07:00
Chris Lu
d30483d642 re-enable system logs 2020-04-12 14:03:07 -07:00
Chris Lu
6f948e4887 remove configurable topics folder location 2020-04-12 13:07:59 -07:00
Chris Lu
94e35cdb35 mount: fix 
fix bug found by git bisect, but I do not understand why it can cause error!
 2020-04-12 03:34:36 -07:00
Chris Lu
1c2e920fa2 fix compilation error 2020-04-12 03:11:19 -07:00
Chris Lu
b5a713be68 filer store: add redis2 redis_cluster2 as default redis store 2020-04-12 02:50:41 -07:00
Chris Lu
e613695af7 a little optimization 2020-04-12 01:13:57 -07:00
Chris Lu
2a1f396df5 avoid duplicated setting chunks into cache 2020-04-12 01:06:50 -07:00
Chris Lu
b9b7da905e handle nil chunk cache 2020-04-12 01:00:12 -07:00
Chris Lu
211d87cf4c mount: option to disable caching 2020-04-12 00:52:54 -07:00
Chris Lu
1c65656fb4 s3: add option to fsync buckets 2020-04-11 23:37:10 -07:00
Chris Lu
e4af63a721 volume server: accept fsync=true in write requests 2020-04-11 21:39:16 -07:00
Chris Lu
5987810e5e set default chunk size to 16 2020-04-11 21:22:52 -07:00
Chris Lu
df97da25f9 mount: add on disk caching 2020-04-11 21:12:41 -07:00
Chris Lu
c8ca234773 refactoring 2020-04-11 14:27:25 -07:00
Chris Lu
df9d538044 rename function 2020-04-11 14:19:44 -07:00
Chris Lu
d7f3acb2c0 refactor 2020-04-11 12:45:24 -07:00
Chris Lu
ed54682598 refactoring 2020-04-11 12:43:17 -07:00
Chris Lu
e909b55633 refactoring 2020-04-11 12:39:46 -07:00
Chris Lu
b7f0ba3800 refactoring 2020-04-11 12:37:41 -07:00
Chris Lu
417125457e add additional buffers for logs 2020-04-11 02:19:48 -07:00
Chris Lu
4b27afe176 fix sqs message reading 2020-04-10 15:27:47 -07:00
Chris Lu
dce2702e1b fix the instruction 2020-04-10 14:50:10 -07:00
Chris Lu
13ab1fdaf0 mount: use file default permission 0666 2020-04-10 02:23:23 -07:00
Chris Lu
1101a42e5c mv : create new folder, move children, and delete old folder 2020-04-10 01:37:03 -07:00
Chris Lu
bcf37346ef add timestamp inside lock 2020-04-10 01:35:59 -07:00
Chris Lu
9fa065f600 typo in logs 2020-04-09 23:43:09 -07:00
Chris Lu
8db2120bee add logs 2020-04-09 23:42:59 -07:00
Chris Lu
43c62a8a95 1.73 2020-04-09 21:44:30 -07:00
Chris Lu
eb39df2704 avoid dead loop 2020-04-09 21:43:26 -07:00
Chris Lu
56ec8c4513 fix to avoid nil file an dirty pages 2020-04-09 21:43:05 -07:00
Chris Lu
59f40e2027 volume: best effort to detect ip address 
fix https://github.com/chrislusf/seaweedfs/issues/1264
 2020-04-09 00:26:24 -07:00
Chris Lu
f6a7e79dc3 weed shell: simplify CLI option for filer 2020-04-08 23:57:15 -07:00
Chris Lu
2329d9e0c1 add volume.fix.replication to default scaffold 2020-04-08 23:36:22 -07:00
Chris Lu
bb78ab9915 wait for master to be aware of the failed volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1268
 2020-04-08 23:12:37 -07:00
Chris Lu
8764bdb9df 1.72 2020-04-08 22:35:14 -07:00
Chris Lu
8a73276602 refactor a bit 2020-04-08 22:33:36 -07:00
Chris Lu
006f78b70b make a copy of request data 
to avoid concurrency issues
 2020-04-08 22:31:19 -07:00
Chris Lu
00b993a234 add util for md5 2020-04-08 22:30:38 -07:00
Chris Lu
ed98223b08 still log, but not persisting the changes 2020-04-08 20:32:57 -07:00
Chris Lu
6630541399 ensure correct file size when opening existing files 2020-04-08 12:50:59 -07:00
Chris Lu
b524a40375 add locking to fs cache 2020-04-08 12:50:34 -07:00
Chris Lu
bd56172b82 simplify file handle management 2020-04-08 12:50:20 -07:00
Chris Lu
dc08e4098f add etag only for PUT or large chunked uploads 2020-04-08 09:13:26 -07:00
Chris Lu
ec2eb8bc48 add If-None-Match and If-Modified-Since 
fix https://github.com/chrislusf/seaweedfs/issues/1269
 2020-04-08 08:12:00 -07:00
Chris Lu
4c498e73de filer replication: add s3 endpoint 2020-04-07 17:49:00 -07:00
Chris Lu
745f5d2a25 fix setting default value 2020-04-07 01:58:48 -07:00
Chris Lu
54debdc6f7 filer: fix configuration settings 2020-04-07 01:30:53 -07:00
Chris Lu
90d576affe disable meta data change event logging for now. 2020-04-07 01:05:24 -07:00
Chris Lu
b51d8f03bb 1.71 2020-04-05 22:04:16 -07:00
Chris Lu
5a5908407d filer: support larger file size 
fix https://github.com/chrislusf/seaweedfs/issues/1257
 2020-04-05 21:48:45 -07:00
Chris Lu
19edd9c091 comment out debug messages 2020-04-05 20:15:17 -07:00
Chris Lu
78ea937606 fix 2020-04-05 17:34:26 -07:00
Chris Lu
53626734d4 iterate through the log buffer 2020-04-05 16:51:30 -07:00
Chris Lu
f71b855db5 adjust caching and notification for recursive directory deletion 2020-04-05 15:03:25 -07:00
Chris Lu
f14b6a09a8 remove duplicated update events 2020-04-05 13:22:15 -07:00
Chris Lu
91da7057b1 refactoring 2020-04-05 13:11:43 -07:00
Chris Lu
2d43f85577 watch entries with common path prefix 2020-04-05 12:51:21 -07:00
Chris Lu
7bc3c93512 add util.PathJoin 2020-04-05 12:40:46 -07:00
Chris Lu
af1f64d244 change from 'weed tail' to 'weed watch' 2020-04-05 12:39:20 -07:00
Chris Lu
b10679fcf0 adjust log file names 2020-04-05 12:38:46 -07:00
Chris Lu
2cb1700ecb set to 4MB meta log buffer size 2020-04-05 12:38:29 -07:00
Chris Lu
e78ee087e9 fix tests 2020-04-05 01:17:35 -07:00
Chris Lu
dcc6b6a3ed go fmt 2020-04-05 00:52:31 -07:00
Chris Lu
bf270d9e8c filer: able to tail meta data changes 2020-04-05 00:51:16 -07:00
Chris Lu
f7b5a1d697 add disk stat to logs 2020-04-04 13:17:23 -07:00
Chris Lu
6e776aaafc loop through all masters 2020-04-03 00:47:48 -07:00
Chris Lu
4a5e381feb fail fast if filer configured wrong masters 
fix https://github.com/chrislusf/seaweedfs/issues/1257
 2020-04-03 00:47:33 -07:00
Chris Lu
aaf5e658cc adjust parameter name 2020-04-03 00:41:05 -07:00
Chris Lu
6c96e51414 re-ordering 2020-04-03 00:40:54 -07:00
Chris Lu
d2f8e50459 make error a little bit more clearer 
fix https://github.com/chrislusf/seaweedfs/issues/1259
 2020-04-03 00:24:09 -07:00
Chris Lu
5f881d2fd5 mount: fail early rather than after mounted the directory 
fix https://github.com/chrislusf/seaweedfs/issues/1258
 2020-04-03 00:21:03 -07:00
Chris Lu
fee3319cbf handle gzip for image resizing 2020-04-02 23:44:24 -07:00
Chris Lu
73bc286377 volume: passing ext info via URL 2020-04-02 21:23:15 -07:00
Chris Lu
cea52a4faf volume copying adds cleaning up on error 
fix https://github.com/chrislusf/seaweedfs/issues/1253
 2020-04-02 15:36:55 -07:00
Chris Lu
c981975627 volume.fix.replication: add test cases for complicated moving 
fix https://github.com/chrislusf/seaweedfs/issues/1253
 2020-04-02 02:16:16 -07:00
Chris Lu
ae3d08883f volume.fsck: fix for erasure encoded volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1254
 2020-04-01 22:10:13 -07:00
Chris Lu
c446438ca5
Merge pull request #1255 from levenlabs/ignore 
Added treat_replication_as_minimums master toml option
 2020-04-01 16:48:58 -07:00
James Hartig
eae3f27c80 Added treat_replication_as_minimums master toml option 2020-04-01 19:08:48 -04:00
Chris Lu
2fc2eb74dd refactoring 2020-03-30 22:57:45 -07:00
Chris Lu
c22094d340 adjust logging 2020-03-30 13:03:43 -07:00
Chris Lu
50a5018b7f writing meta logs is working 2020-03-30 01:19:33 -07:00
Chris Lu
9dc0b1df8f refactoring to compile for windows OS 
windows os does not like to work with fuse
 2020-03-29 21:07:55 -07:00
Chris Lu
be6c2f8511 1.70 2020-03-29 13:40:55 -07:00
Chris Lu
54768d0761 fix: delete a file and then create a directory with the same name 2020-03-29 01:39:48 -07:00
Chris Lu
057722bbf4 return part of the chunk if chunkview is not the full chunk 2020-03-29 00:54:39 -07:00
Chris Lu
af6d9976ad adjust print out 2020-03-28 14:07:25 -07:00
Chris Lu
a75d50bbb8 FUSE: add configurable in memory chunk cache size 2020-03-28 14:07:16 -07:00
Chris Lu
826bc0b7e3 FUSE: add chunk cache for recently accessed file chunks 2020-03-28 13:43:31 -07:00
Chris Lu
4aa82c95e6 refactor 2020-03-28 13:42:35 -07:00
Chris Lu
eedd33dda3 refactoring 2020-03-28 13:41:58 -07:00
Chris Lu
005b4ab3fe 1.69 2020-03-27 04:51:31 -07:00
Chris Lu
d1439c5bd3 fix FUSE read for large files 
FUSE expects ReadAt do not return partial filled buffer with a nil error.
 2020-03-27 04:50:51 -07:00
Chris Lu
f06ca04451 avoid overflow 2020-03-27 04:35:31 -07:00
Chris Lu
e1911760a7 refactoring 2020-03-27 00:30:55 -07:00
Chris Lu
2f5ed29c37 clean up 2020-03-27 00:30:39 -07:00
Chris Lu
2286eda575 sampling whether the data can be gzipped 2020-03-26 23:50:48 -07:00
Chris Lu
5463560912 small test 2020-03-26 23:50:07 -07:00
Chris Lu
34a0d424f8 FUSE mount: report max volume count available disk space 2020-03-26 23:06:21 -07:00
Chris Lu
a5ce359999 refactoring 2020-03-26 22:46:49 -07:00
Chris Lu
0a05052b5c 1.68 2020-03-26 22:22:36 -07:00
Chris Lu
e7e8691720 add some tests 2020-03-26 22:21:52 -07:00
Chris Lu
86398d1efe S3 API: set deletion to recursive 
fix https://github.com/chrislusf/seaweedfs/issues/1251
 2020-03-26 11:05:54 -07:00
Chris Lu
1fef598347 set target Dir object when moving 2020-03-26 10:56:40 -07:00
Chris Lu
9e2e07b238 tweak 2020-03-26 10:56:18 -07:00
Chris Lu
cbce793753 rename: set new name for renamed files 2020-03-26 03:30:23 -07:00
Chris Lu
c763a492f5 refactor 2020-03-26 03:30:02 -07:00
Chris Lu
712720864d error checking for remove 2020-03-26 02:21:30 -07:00
Chris Lu
8d9a76005d add back running output 2020-03-26 02:14:48 -07:00
Chris Lu
dd5b582d05 go fmt 2020-03-26 00:09:01 -07:00
Chris Lu
7f0d87b206 tree structured fs cache 
FsCache for FsNode, wrapping fs.Node
 2020-03-26 00:08:14 -07:00
Chris Lu
2e4fadd10a fix fscache move 2020-03-25 22:19:19 -07:00
Chris Lu
966df2ae27 purging skip EC volumes 2020-03-25 02:41:22 -07:00
Chris Lu
6d87a0c5f5 batch delete orphan data by file keys 2020-03-25 02:21:15 -07:00
Chris Lu
5ebc95b69b refactoring 2020-03-25 02:20:19 -07:00
Chris Lu
d79f0a76b1 format output 2020-03-25 00:56:47 -07:00
Chris Lu
d9b56ed88b shell: volume.fsck add options 2020-03-24 22:38:36 -07:00
Chris Lu
e63a79ade8 better handle lock in case of exception 2020-03-24 18:41:25 -07:00
Chris Lu
4d5554b16f fix compilation on travis 2020-03-24 18:40:52 -07:00
Chris Lu
d83e648c0b auto add prefix "fs." 2020-03-24 02:40:51 -07:00
Chris Lu
a875f67a08 fix pwd printout 2020-03-24 02:40:29 -07:00
Chris Lu
00d3f01876 better output format 2020-03-24 02:34:28 -07:00
Chris Lu
0820935290 clean up 2020-03-24 02:20:37 -07:00
Chris Lu
49440cbd2f shell: add volume.fsck 
volume.fsck can find out how much space is not used by the filer
 2020-03-24 02:18:13 -07:00
Chris Lu
0f10833e0b fix compilation 2020-03-24 00:16:12 -07:00
Chris Lu
3794f2721f fix logic 2020-03-24 00:09:12 -07:00
Chris Lu
443b229ef7 disable chunk file output 2020-03-24 00:08:02 -07:00
Chris Lu
bb9b97e2b4 add comments 2020-03-23 23:07:11 -07:00
Chris Lu
8047ec2f51 shell: fs.meta.save add option to export all fileIds for all files 2020-03-23 22:54:46 -07:00
Chris Lu
782d776d2a refactoring 2020-03-23 22:54:02 -07:00
Chris Lu
38e73463f1 fix du block and byte couting 2020-03-23 21:37:04 -07:00
Chris Lu
b51fa81f0e fix directory checking 2020-03-23 21:36:39 -07:00
Chris Lu
e666aeece2 simplify parsing filer host and port 2020-03-23 21:26:15 -07:00
Chris Lu
d151185b7e shell: desupport filer url in the arguments 2020-03-23 20:46:17 -07:00
Chris Lu
40601953bf comments 2020-03-23 02:54:49 -07:00
Chris Lu
daeb18be30 scaffold: fix master.toml typo 
fix https://github.com/chrislusf/seaweedfs/issues/1247
 2020-03-23 01:45:43 -07:00
Chris Lu
b97768c51c refactoring 2020-03-23 01:30:22 -07:00
Chris Lu
45ee3736aa refactoring 2020-03-23 01:25:38 -07:00
Chris Lu
cddcb5cafe same booleans as before 2020-03-23 01:25:23 -07:00
Chris Lu
f84c288852 purge 2020-03-23 01:14:55 -07:00
Chris Lu
a5aa8be19b Update webdav_server.go 2020-03-23 01:14:42 -07:00
Chris Lu
7f1e3c843d refactoring 2020-03-23 01:14:21 -07:00
Chris Lu
c07bcd5065 refactoring 2020-03-23 00:30:02 -07:00
Chris Lu
654a69ff52 refactoring 2020-03-23 00:06:24 -07:00
Chris Lu
c0f0fdb3ba refactoring 2020-03-23 00:01:34 -07:00
Chris Lu
fbca6b29bd refactoring 2020-03-22 23:52:55 -07:00
Chris Lu
e39e78ea8d remove println 2020-03-22 18:37:12 -07:00
Chris Lu
c16dc57a58 fix max volume count reporting 2020-03-22 18:33:19 -07:00
Chris Lu
d6412f27a0 logging 2020-03-22 18:32:56 -07:00
Chris Lu
35208711e5 logging 2020-03-22 18:32:49 -07:00
Chris Lu
e32999108a add auto configured volume max count help messge 2020-03-22 16:27:15 -07:00
Chris Lu
3137777d83 volume: automatically detect max volume count 2020-03-22 16:21:42 -07:00
Chris Lu
0bf148f49d logging 2020-03-22 16:19:00 -07:00
Chris Lu
bda13ed593 commented prints 2020-03-22 13:09:51 -07:00
Chris Lu
74e23e8d64 FUSE mount: rename correctly across folders 2020-03-22 13:09:33 -07:00
Chris Lu
64000f5c45 FUSE mount: rename under root 
fix https://github.com/chrislusf/seaweedfs/issues/1242
 2020-03-22 13:09:16 -07:00
Chris Lu
9f2c9b666b adjust log level 2020-03-22 01:39:08 -07:00
Chris Lu
ae2ee379c0 consistent 64bit size 2020-03-22 01:37:46 -07:00
Chris Lu
2bdd936fb6 purge code 2020-03-22 01:33:40 -07:00
Chris Lu
e93e986727 webdav: stream read 2020-03-22 01:30:27 -07:00
Chris Lu
65d2ea9fb0 FUSE mount: stream read data with buffer 
fix https://github.com/chrislusf/seaweedfs/issues/1244
 2020-03-22 01:00:36 -07:00
Chris Lu
82bfad5b86 refactoring 2020-03-21 23:48:11 -07:00
Chris Lu
6ff9e2835e filer: skip resizing image if width or height larger than original image 
fix https://github.com/chrislusf/seaweedfs/issues/1239#issuecomment-602140779
 2020-03-21 22:16:00 -07:00
Chris Lu
7c111f7b75 FUSE mount: make "nonempty" optional 
https://github.com/chrislusf/seaweedfs/issues/1094
 2020-03-21 19:14:25 -07:00
Chris Lu
3b3e063f25 1.67 2020-03-20 23:42:49 -07:00
Chris Lu
d3e4a31058 filer: fix where deletion can miss under high concurrency 2020-03-20 23:39:32 -07:00
Chris Lu
cbfe31a9a8 idx file sync before compaction 2020-03-20 23:38:46 -07:00
Chris Lu
3505b06023 report deletion error in the log 2020-03-20 23:38:24 -07:00
Chris Lu
4bc19f1e52 master: fix connections count reporting 2020-03-20 22:41:37 -07:00
Chris Lu
e1b5ac5c04 1.66 2020-03-20 20:39:48 -07:00
Chris Lu
308688c8d0 filer: add back image resizing capability 2020-03-20 20:31:11 -07:00
Chris Lu
f251d03673 refactoring 2020-03-20 15:46:16 -07:00
Chris Lu
11372dd548 refactoring 2020-03-20 15:38:04 -07:00
Chris Lu
c4bea45099 S3 API: fix DeleteMultipleObjectsHandler 
fix https://github.com/chrislusf/seaweedfs/issues/1241
 2020-03-20 14:17:31 -07:00
Chris Lu
165b0d22a4 1.65 2020-03-20 00:05:42 -07:00
Chris Lu
81797a059a volume: sync volume file right before compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1237
 2020-03-19 23:54:52 -07:00
Chris Lu
709f231e23 tiered storage: add s3 endpoint for private s3 implementation 
fix https://github.com/chrislusf/seaweedfs/issues/1238
 2020-03-19 21:13:56 -07:00
Chris Lu
d848d08944 use public url also for deletions 2020-03-18 11:16:45 -07:00
Chris Lu
6566c8e114 weed upload: add usePublicUrl option 
fix https://github.com/chrislusf/seaweedfs/issues/1236
 2020-03-18 10:50:53 -07:00