mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-27 03:17:55 +08:00
Code Issues Packages Projects Releases Wiki Activity
2,529 Commits 32 Branches 296 Tags 153 MiB
6a573d62e0
Commit Graph

1211 Commits

Author SHA1 Message Date
Chris Lu
104922a3db text wrapping 2019-03-18 00:35:15 -07:00
Chris Lu
44647a46c0 needle scanner read appendAtNs 2019-03-17 23:28:43 -07:00
Chris Lu
8b26d15740 refactoring: simplify function parameter 2019-03-17 22:32:01 -07:00
Chris Lu
2fcc88116e go fmt 2019-03-17 20:27:36 -07:00
Chris Lu
aca653c08b weed shell: list volumes 2019-03-17 20:27:08 -07:00
Chris Lu
22fbbf023b adjust collection list output 2019-03-17 17:31:05 -07:00
Chris Lu
e48267e287 adjust output 2019-03-17 17:28:29 -07:00
Chris Lu
657dd2e6c9 add shell command to list all collections 2019-03-16 13:43:16 -07:00
Chris Lu
b92122b885 fix test 2019-03-16 09:50:57 -07:00
Chris Lu
36b632ebcb fix tests 2019-03-16 09:50:21 -07:00
Chris Lu
42cb05c088 add context.Context 2019-03-15 17:23:02 -07:00
Chris Lu
55bab1b456 add context.Context 2019-03-15 17:20:24 -07:00
Chris Lu
cece860bfd add context to all filer APIs 2019-03-15 15:55:34 -07:00
Chris Lu
55c85f3a66 adding context 2019-03-15 15:26:09 -07:00
Chris Lu
d30600e6d6 fix filer starting under "weed server" when ip is not localhost 
fix https://github.com/chrislusf/seaweedfs/issues/886
 2019-03-15 09:57:33 -07:00
Chris Lu
a4e8aa98a0 weed server: mkdir first, and set the master ip address 
fix https://github.com/chrislusf/seaweedfs/issues/885
 2019-03-15 00:32:42 -07:00
Chris Lu
95ef4513c8 print out memory size 2019-03-13 23:14:40 -07:00
Chris Lu
346541a101 print out per entry memory usage 2019-03-13 23:07:24 -07:00
Chris Lu
99ce10daeb Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-03-11 00:58:17 -07:00
Chris Lu
2ce4dae795 move to 1.26 2019-03-11 00:58:12 -07:00
chenwanli
c668e55e07 Revert "fix https://github.com/chrislusf/seaweedfs/issues/780" 
This reverts commit 31038acccf.
 2019-03-11 11:48:11 +08:00
Chris Lu
6fe071175d mount: add option to disable allow others 
fix https://github.com/chrislusf/seaweedfs/issues/877
 2019-03-10 19:45:40 -07:00
Chris Lu
ad08a52ab6 synchronously upload files 
fix https://github.com/chrislusf/seaweedfs/issues/807
 2019-03-09 13:24:22 -08:00
Chris Lu
4773497d2c add back "/cluster/status" 
fix https://github.com/chrislusf/seaweedfs/issues/870
 2019-03-06 12:10:45 -08:00
Chris Lu
ab32556a51 weed server: fix error coming from moving raft to grpc 
fix https://github.com/chrislusf/seaweedfs/issues/872
 2019-03-06 10:21:49 -08:00
Chris Lu
0898e6c38e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-03-04 13:00:10 -08:00
Chris Lu
d312c55bbe file path supports windows, avoiding back slashes 
fix https://github.com/chrislusf/seaweedfs/issues/868
 2019-03-04 13:00:08 -08:00
Chris Lu
f9dcf56465 adjust log level for loading configurations 2019-03-04 09:02:58 -08:00
chenwanli
f233bbe9c0 Fix https://github.com/chrislusf/seaweedfs/issues/825#issuecomment-469097375 2019-03-04 12:43:43 +08:00
Chris Lu
219b651bc3 jwt check the base file id 
fix https://github.com/chrislusf/seaweedfs/issues/867
 2019-03-03 10:17:44 -08:00
Chris Lu
2812c14520 master: add option to disable http operations 2019-03-02 05:10:05 -08:00
chenwanli
900aad4935 Fix https://github.com/chrislusf/seaweedfs/issues/864 2019-02-28 11:37:44 +08:00
Chris Lu
cb07d15254 add namespace for s3 2019-02-27 00:21:37 -08:00
Chris Lu
70ac2f6ea8 add namespace for ListAllMyBucketsResult 2019-02-26 23:23:04 -08:00
chenwanli
fd27ed7755 Fix https://github.com/chrislusf/seaweedfs/issues/861 2019-02-26 17:12:39 +08:00
Chris Lu
344caf3cd7 1.25 2019-02-25 00:44:48 -08:00
Chris Lu
7bbe24dd28 volume server directly support https 2019-02-25 00:43:36 -08:00
Chris Lu
c892b898a1 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-02-25 00:34:25 -08:00
Chris Lu
0d84e32f56 update library versions 2019-02-25 00:34:21 -08:00
chenwanli
0bc3cdd26a Fix https://github.com/chrislusf/seaweedfs/issues/825 2019-02-25 10:47:41 +08:00
Chris Lu
0a106c1757 updated glide 2019-02-20 09:09:47 -08:00
Chris Lu
e108688990 avoid grpc 5 seconds timeout 
some operations may take longer than 5 seconds.

only keep the timeout for raft operations
 2019-02-20 01:01:01 -08:00
Chris Lu
0fa1be8c4b adjust formatting and comments 2019-02-19 21:10:10 -08:00
Chris Lu
72c2e6bb8f remove unused file 2019-02-19 14:16:19 -08:00
Chris Lu
07af52cb6f raft change from http to grpc 
master grpc port is fixed to http port + 10000
 2019-02-18 22:38:14 -08:00
Chris Lu
448645203a remove unused variables 2019-02-18 20:05:55 -08:00
Chris Lu
d37c3ab7a5 adjust error message 2019-02-18 18:03:27 -08:00
Chris Lu
2442d56671 adjust package name 2019-02-18 18:03:16 -08:00
Chris Lu
9a4dda3011
Merge pull request #855 from chrislusf/add_jwt 
Add jwt
 2019-02-18 15:05:32 -08:00
Chris Lu
a1c7dc3806 avoid changing attributes for mount directory 2019-02-18 12:14:28 -08:00
Chris Lu
77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu
55761ae806 skip nil options 2019-02-16 12:49:58 -08:00
Chris Lu
98a03b38e5 avoid util package depends on security package 2019-02-16 12:23:35 -08:00
Chris Lu
31aa60ce05 Merge branch 'master' into add_jwt 2019-02-16 08:55:38 -08:00
Chris Lu
ad257ae179 simplify isLeader() logic 2019-02-16 08:55:23 -08:00
Chris Lu
712a0e19c8 Merge branch 'master' into add_jwt 2019-02-15 10:01:16 -08:00
Chris Lu
2ec6a679c2 avoid "fchmod failed" during cp 2019-02-15 10:00:27 -08:00
Chris Lu
157c0f7c01 add jwt for writes 2019-02-15 09:59:22 -08:00
Chris Lu
7103c1ab7e go fmt 2019-02-15 00:09:48 -08:00
Chris Lu
74fb237727 benchmark can work in secure mode 2019-02-15 00:09:19 -08:00
Chris Lu
a3b0e39b06 weed mount deletion always goes to weed filer 2019-02-14 07:07:38 -08:00
Chris Lu
215cd27b37 add authorizing fileId write access 
need to secure upload/update/delete for benchmark/filer/mount
need to add secure grpc
 2019-02-14 00:08:20 -08:00
Chris Lu
4ff4a147b2 cleanup security.Secret 2019-02-09 21:56:32 -08:00
Chris Lu
501bd72b1c wip: add security.toml file 2019-02-09 21:07:12 -08:00
Chris Lu
729d71fd4c Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-02-09 12:52:12 -08:00
Chris Lu
8afc632484 raft: use the first master to bootstrap the election 2019-02-09 12:52:09 -08:00
Chris Lu
744abc2690 log raft state changes 2019-02-09 12:51:14 -08:00
Sergey
aa5ccff6d2
fixing of typos 2019-02-06 18:59:15 +05:00
Chris Lu
1334507595 Revert "randomize based on self address" 
This reverts commit 6230eb28a6.
 2019-01-28 12:12:51 -08:00
Chris Lu
6230eb28a6 randomize based on self address 
fix #851
 2019-01-28 11:55:33 -08:00
Chris Lu
221105eea3 Revert "use the first entry to bootstrap master cluster" 
This reverts commit 40c8725ffa.
 2019-01-28 11:46:46 -08:00
Chris Lu
7a493bbefa better way to detect a master isLeader 2019-01-28 10:36:16 -08:00
Chris Lu
40c8725ffa use the first entry to bootstrap master cluster 
fix https://github.com/chrislusf/seaweedfs/issues/851
 2019-01-28 10:35:28 -08:00
Chris Lu
adcfe66034 1.24 2019-01-26 00:15:50 -06:00
Chris Lu
834f414af9 add a timeout 2019-01-26 00:15:42 -06:00
Chris Lu
09471b46f9 fix related tests 2019-01-26 00:10:29 -06:00
Chris Lu
5668ed798d memory needle map mark size to be TombstoneFileSize 
fix https://github.com/chrislusf/seaweedfs/issues/850
 2019-01-26 00:05:44 -06:00
Chris Lu
d3f675abc6 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-01-24 22:39:41 -08:00
Chris Lu
e70f740deb set file attributes 
fix https://github.com/chrislusf/seaweedfs/issues/847
 2019-01-24 19:38:04 -08:00
bingoohuang
cf0fc797ac use tempFile.Truncate to reset file size to speedup test 2019-01-23 16:24:51 +08:00
Chris Lu
3f56b12ed4 raft: adding idle connection time out 
another attempt to fix https://github.com/chrislusf/seaweedfs/issues/825
 2019-01-22 09:25:25 -08:00
chenwanli
f47f8be912 Fix block-waiting for leader when there are no volumes 2019-01-22 15:58:37 +08:00
Chris Lu
8ec57e18c7 1.23 2019-01-18 14:27:34 -08:00
Chris Lu
a6bffc58f9 go fmt 2019-01-18 14:15:10 -08:00
Chris Lu
67e2ea72be master add separate grpc port 
due to https://github.com/soheilhy/cmux/issues/64

fix https://github.com/chrislusf/seaweedfs/issues/820
fix https://github.com/chrislusf/seaweedfs/issues/840
fix https://github.com/chrislusf/seaweedfs/issues/841
 2019-01-18 14:14:47 -08:00
Chris Lu
dfae0f4e9d 1.22 2019-01-18 00:25:20 -08:00
Chris Lu
e84dcb0bfe fix timeout issue, but can not reproduce with local built binaries 
fix https://github.com/chrislusf/seaweedfs/issues/841
 2019-01-18 00:24:40 -08:00
Chris Lu
1d103e3ed5 timeout http connections 
possible fix for https://github.com/chrislusf/seaweedfs/issues/825
 2019-01-17 23:38:33 -08:00
Chris Lu
b881e11a16 1.21 2019-01-16 20:44:38 -08:00
bingoohuang
ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
bingoohuang
6ddfaf33cb extract VolumeFileScanner for ScanVolumeFile 2019-01-16 17:48:59 +08:00
bingoohuang
27093bc2e5 fix needle Append return offset to avoid uint32 overflow 2019-01-16 16:48:03 +08:00
bingoohuang
219838db87
fail fast when failed to write local volume 2019-01-15 15:47:43 +08:00
Chris Lu
2ff95ead57
Merge pull request #832 from PapaYofen/fix-830 
Fix https://github.com/chrislusf/seaweedfs/issues/830
 2019-01-12 17:26:51 -08:00
chenwanli
b71d6f4b88 Fix https://github.com/chrislusf/seaweedfs/issues/830 2019-01-12 10:07:19 +08:00
chenwanli
80cf99db1b Set rpc dial timeout to 5s 2019-01-11 18:23:31 +08:00
Chris Lu
8613e4f558
Merge pull request #827 from suilongfei/master 
change log_dir to logdir for the "flag redefined:log_dir" error when use another package with "github.com/golang/glog"
 2019-01-10 09:10:07 -08:00
Chris Lu
2d23d86fd3 no timeout for volume vacuum 
revert changes on volume vacuum timemout from https://github.com/chrislusf/seaweedfs/pull/829
 2019-01-10 09:07:40 -08:00
chenwanli
39c7455881 Set http timeout to 5s 2019-01-10 20:21:08 +08:00
chenwanli
0a3e83a36a Set timeout for master and volume non-streaming rpc 2019-01-10 19:41:03 +08:00
隋龙飞@视频结构化
b864cf7e07
change log_dir flag to logdir 
change log_dir flag to logdir
 2019-01-10 17:17:32 +08:00
隋龙飞@视频结构化
ea4f29d1e7
change log_dir flag to logdir 
change log_dir flag to logdir
 2019-01-10 17:16:58 +08:00
Chris Lu
2a75a36b27 set cassandra ttl with entry.TTLSec 2019-01-08 20:26:19 -08:00
byungjo.yoon
58fd2e5a19 Set redis key ttl with entry.TTLSec 2019-01-09 11:48:19 +09:00
Chris Lu
badd8fdf8f shortcut for empty file 2019-01-08 09:03:28 -08:00
chenwanli
41fe31b5d0 Speed up heartbeat to new master leader 2019-01-07 16:09:15 +08:00
Chris Lu
e768e64fbc 1.20 2019-01-06 10:52:26 -08:00
Chris Lu
bd32108a90 disable async file deletion 2019-01-05 22:51:50 -08:00
Chris Lu
be9a7592a1 incrementally calculate visible intervals 2019-01-05 22:23:44 -08:00
Chris Lu
6b5d6bb5a6 adjust error names 2019-01-05 19:52:38 -08:00
Chris Lu
9383c91eb1 wait to read again if the volume is compacting 2019-01-05 19:52:17 -08:00
Chris Lu
fe50224ea0 refactoring 2019-01-05 15:21:56 -08:00
Chris Lu
6740427638 weed mount: reduce default chunk size 2019-01-05 15:17:03 -08:00
Chris Lu
d9f9aca453 refactoring 2019-01-05 15:16:39 -08:00
Chris Lu
251b596bc5 increase batch deletion size to 1024 2019-01-03 22:21:57 -08:00
Chris Lu
aac11fb832 reduce directory list cache usage 2019-01-03 22:21:39 -08:00
Chris Lu
7f74577db1 add Last-Modified to HEAD requests 
to work better with restic
 2019-01-02 14:23:25 -08:00
Chris Lu
ceafb7d145 1.19 2019-01-02 12:59:20 -08:00
Chris Lu
28a41fda3e assign new file id even on filer PUT operations 2019-01-02 12:58:26 -08:00
Chris Lu
d67f7ddfaf adjust log 2019-01-02 12:58:06 -08:00
Chris Lu
e49a38842a set etag in response 2019-01-02 12:57:54 -08:00
Chris Lu
d5ebfd5b0c fix s3cmd put 
1. fix volume put data size
 2019-01-02 12:57:33 -08:00
Chris Lu
3339325334 fix s3 ListAllMyBucketsResult to work with s3cmd 2019-01-02 11:36:29 -08:00
Chris Lu
43db7ac123 fix spelling 2019-01-02 02:38:32 -08:00
Chris Lu
9539823c35 fix log 2019-01-01 03:04:44 -08:00
Chris Lu
525db94d9a async file chunk deletion 2019-01-01 02:33:57 -08:00
Chris Lu
ffb5d3f93a get bytes from sync pool 2019-01-01 02:14:40 -08:00
Chris Lu
832ac8129a reduce pointer usage 2018-12-31 15:10:14 -08:00
Chris Lu
1478d7ea21 reduce file seek when writing 2018-12-31 15:08:32 -08:00
Chris Lu
81904ad336 debugging: track all memory allocations 2018-12-31 14:54:13 -08:00
Chris Lu
03cfb4267f adjust vaccum logging 2018-12-31 00:06:52 -08:00
Chris Lu
d6181076bb FUSE remove adjust error 2018-12-30 22:29:27 -08:00
Chris Lu
16e8610426 1.18 2018-12-30 00:52:20 -08:00
Chris Lu
0388d421d2 caching visible intervals for read 
speeds up 4x in single thread mode
speeds up 30% in 32 threads mode
 2018-12-30 00:51:44 -08:00
Chris Lu
55f7d8f801 remove extra log 2018-12-29 18:00:19 -08:00
Chris Lu
170ed53da5 weed mount: mac add AutoXattr 2018-12-29 13:55:38 -08:00
Chris Lu
d8c7844d2e 1.17 2018-12-29 13:16:38 -08:00
Chris Lu
9f3836072a weed mount: no mac finder broswe mode 2018-12-29 13:16:23 -08:00
Chris Lu
d7bf6c5b91 switching_to_seaweedfs_fuse 2018-12-29 10:31:36 -08:00
Chris Lu
0c932d1738 add volume delete grpc API 
fix https://github.com/chrislusf/seaweedfs/issues/808
 2018-12-29 00:03:30 -08:00
Chris Lu
03c7447ad6 set mount point to existing permissions 
fix https://github.com/chrislusf/seaweedfs/issues/806
 2018-12-28 23:36:13 -08:00
Chris Lu
a8af1e3f5f set attributes include crtime 2018-12-28 22:37:18 -08:00
Chris Lu
ba9b7b62d7 1.16 2018-12-28 15:10:49 -08:00
Chris Lu
79168ab0bb adjust error and log 2018-12-28 15:10:23 -08:00
Chris Lu
308ac1d0d2 reuse buffer by sync pool 2018-12-28 03:27:48 -08:00
Chris Lu
260b038441 fix url 2018-12-27 14:43:36 -08:00
Chris Lu
2a19511ba8 avoid gzip benchmark data 
related to https://github.com/chrislusf/seaweedfs/issues/804
 2018-12-27 13:10:46 -08:00
Chris Lu
d3660ffce8 avoid gzipping files without file extensions 
fix https://github.com/chrislusf/seaweedfs/issues/804
 2018-12-27 12:17:05 -08:00
Chris Lu
253f190f48 weed mount add symlink support 2018-12-25 22:45:44 -08:00
Chris Lu
9eb2bee86a avoid error log if not found 2018-12-24 17:45:26 -08:00
Chris Lu
ba923e2864 error if the last arguement does not end with forward slash 2018-12-23 18:20:11 -08:00
Chris Lu
332668a3fd 1.15 2018-12-23 12:54:41 -08:00
Chris Lu
c043fd17cb detect more gzippable content 2018-12-22 15:05:31 -08:00
Chris Lu
49f40cd145 adjust help message 2018-12-22 15:05:15 -08:00
Chris Lu
852ee21835 avoid .gz auto decompression 2018-12-22 13:58:16 -08:00
Chris Lu
be946c9e54 filer store original data size instead of data size after gzip 2018-12-22 13:11:07 -08:00
Chris Lu
141d302492 fix binarySearchCompactSection 2018-12-22 11:52:58 -08:00
Chris Lu
d3839fe279 go fmt 2018-12-22 11:11:57 -08:00
Chris Lu
5333f2984a streaming updates for large entries or large index file 
fix https://github.com/chrislusf/seaweedfs/issues/801
 2018-12-22 11:10:08 -08:00
Chris Lu
36d13355bb compact map fix bug when iterating 2018-12-22 11:05:29 -08:00
Chris Lu
f64dc7465b remove index file 
fix https://github.com/chrislusf/seaweedfs/issues/793
 2018-12-17 20:33:32 -08:00
Chris Lu
16634235c1 1.14 and HCFS 1.0.3 2018-12-17 10:03:03 -08:00
Chris Lu
52712782a2 adding link 2018-12-17 00:31:23 -08:00
Chris Lu
2fb740a2b9 fix upload path 2018-12-17 00:21:34 -08:00
Chris Lu
7b7f4215aa filer update existing entry 2018-12-17 00:20:00 -08:00
Chris Lu
6071be104b adjust log format 2018-12-17 00:03:14 -08:00
Chris Lu
3ac54792e1 paginate when filer deleting and FUSE mount renaming 2018-12-16 23:20:08 -08:00
Chris Lu
93d0ad09af error log format 2018-12-16 17:37:10 -08:00
Chris Lu
66d04cb83f 1.12 2018-12-15 08:17:24 -08:00
Chris Lu
42cb9b76c4 reduce needle map memory usage by 25% 2018-12-15 05:55:56 -08:00
Chris Lu
42b8f1145a weed mount ls paginate the results 
fix https://github.com/chrislusf/seaweedfs/issues/787
 2018-12-11 07:41:50 -08:00
Chris Lu
39ea8484c1 adjust proto API 2018-12-11 07:17:10 -08:00
Chris Lu
168f461761 fix MaxUint8 2018-12-10 20:57:51 -08:00
Chris Lu
8f78a069c6 fix possible name overflow when len(name)>255 2018-12-10 20:57:01 -08:00
Chris Lu
4a490fe35a go fmt 2018-12-09 01:27:25 -08:00
Chris Lu
5741134ca0 change overflow from map to slice 2018-12-09 01:27:11 -08:00
Chris Lu
d9189c857c remove unused file 2018-12-09 00:30:53 -08:00
Chris Lu
df4b9df142 add memory usage report 2018-12-09 00:15:23 -08:00
Chris Lu
ec5a305624 refactor 2018-12-09 00:12:37 -08:00
Chris Lu
9d3be33e5c optimize a little bit 2018-12-08 21:45:14 -08:00
Chris Lu
36d1e739e6 1.11 2018-12-08 00:32:34 -08:00
Chris Lu
1bfb96f34d optimization for reading whole chunk with gzip encoding 2018-12-07 01:57:55 -08:00
Chris Lu
29f1673d97 refactoring 2018-12-07 01:25:01 -08:00
Chris Lu
c28e8a2397 refactoring 2018-12-06 00:44:41 -08:00
Chris Lu
ffa2827ab1 fail fast if two notification queues or inputs are enabled 2018-12-06 00:37:59 -08:00
Chris Lu
63619f8b8e fail early if two filer stores are defined 2018-12-06 00:32:36 -08:00
Chris Lu
52b24a9902 default "weed server -filer" to same directory as -mdir 2018-12-05 23:24:25 -08:00
Chris Lu
fb43a9d7b9 adjust message 2018-12-05 02:03:03 -08:00
chenwanli
31038acccf fix https://github.com/chrislusf/seaweedfs/issues/780 2018-12-05 14:06:16 +08:00
Chris Lu
650d1af978 HCFS support change user name and group names 2018-12-04 00:46:00 -08:00
Chris Lu
b0838db5a8 support hdfs user name and group names 2018-12-04 00:13:40 -08:00
Chris Lu
7ace0efd65 revert prev 2018-12-02 23:25:37 -08:00
Chris Lu
738247d47e do not throw error if not found 2018-12-02 23:20:29 -08:00
Chris Lu
c6b646e92c skip creation for root 2018-12-02 23:16:17 -08:00
Chris Lu
bea162c34c add warning for empty attributes 2018-12-02 22:57:59 -08:00
Chris Lu
f0677c5af1 default root to 0755 2018-12-02 19:59:47 -08:00
Chris Lu
db22666a30 add default root directory 2018-12-02 19:42:50 -08:00
Chris Lu
a3def2bbd9 for 1.10 2018-11-29 09:15:04 -08:00
Chris Lu
7a6f49cd42 refactor a bit more 
fix https://github.com/chrislusf/seaweedfs/issues/777
 2018-11-29 00:07:54 -08:00
Chris Lu
ceca078acb avoid overwriting file or directory 
fix https://github.com/chrislusf/seaweedfs/issues/777
 2018-11-29 00:00:56 -08:00
Chris Lu
1cbd53c01c WIP SeaweedFileSystem added mkdirs, getFileStatus, listStatus, delete 2018-11-25 13:43:26 -08:00
Chris Lu
871dee4674 fix test 2018-11-23 12:37:09 -08:00
Chris Lu
86dd933596 go fmt 2018-11-23 00:26:15 -08:00
Chris Lu
444dfded84 add fs.FSStatfser for SeaweedFS weed mount 2018-11-23 00:24:51 -08:00
Chris Lu
2e32b44061 add template for fs.FSStatfser 2018-11-22 23:05:22 -08:00
Chris Lu
6344469a9b set fuse subtype 2018-11-22 23:04:23 -08:00
Chris Lu
d64546c791 adjust logs 2018-11-22 14:31:10 -08:00
Chris Lu
e12b6c0306 add subimit count parameter 
related to https://github.com/chrislusf/seaweedfs/issues/631
 2018-11-22 13:45:29 -08:00
Chris Lu
0ed816d4e9 formatting error log 2018-11-22 00:42:56 -08:00
Chris Lu
19d546938a forgot to upgrade version number 2018-11-21 23:30:59 -08:00
Chris Lu
60d3681371 default images.fix.orientation to false 2018-11-21 16:50:42 -08:00
Chris Lu
99256100a0 fix tests 2018-11-21 16:35:27 -08:00
Chris Lu
e5ad2223a1 add missing intervals 
fix https://github.com/chrislusf/seaweedfs/issues/774
 2018-11-21 16:25:13 -08:00
Chris Lu
c9b3ef58de options to control filer.export verbosity 2018-11-21 15:15:40 -08:00
Chris Lu
2f2e417b68 detailed export 2018-11-21 15:12:47 -08:00
Chris Lu
b282e34dc2 async file chunk deletion 2018-11-20 20:56:28 -08:00
Chris Lu
5065d4ab2d master add grpc API for fileid assigning 2018-11-20 11:35:45 -08:00
Chris Lu
b318167cd3 batch delete file ids 2018-11-20 11:35:08 -08:00
Chris Lu
06474e5990 1.08 2018-11-18 22:43:05 -08:00
Chris Lu
ee3fe07acf fix sort bug 2018-11-18 21:59:53 -08:00
Chris Lu
469863f7f0 merge interval now is quite faster with reusing slices 2018-11-18 21:24:58 -08:00
Chris Lu
d2924c7748 merge intervals is a bit faster 2018-11-18 20:31:39 -08:00
Chris Lu
cb5a10c6a3 adjust the tests 2018-11-18 17:05:36 -08:00
Chris Lu
f1db22d48b add benchmark test for merging intervals 2018-11-18 16:49:08 -08:00
Chris Lu
4fcfc9410f cleanup 2018-11-18 11:51:38 -08:00
Chris Lu
9655dc9ca9 simpler logic 2018-11-18 10:07:30 -08:00
Chris Lu
f8eb988347 1.07 2018-11-18 08:10:15 -08:00
Chris Lu
60421b23f0 reduce weed mount memory usage 2018-11-18 07:49:14 -08:00
Chris Lu
2929563294 1.06 2018-11-14 23:51:15 -08:00
Chris Lu
d3b7965c76 caching grpc clients 2018-11-14 23:50:46 -08:00
Chris Lu
d89774cb7f cache grpc client connections to improve performance 2018-11-14 23:31:39 -08:00
Chris Lu
2e6a3c7b16 reduce memory usage 2018-11-14 22:48:54 -08:00
Chris Lu
f224b9fe28 1.05 2018-11-13 23:54:17 -08:00
Chris Lu
11b0a9cb76 fix possible error when reusing file handles 2018-11-13 23:53:17 -08:00
Chris Lu
a62adf62ca a bit faster 2018-11-12 11:19:13 -08:00
Chris Lu
affe708961 a bit faster 2018-11-11 00:43:30 -08:00
Chris Lu
a09eaf8756 adjusting, seems there are some perf benefit 2018-11-11 00:08:22 -08:00
Chris Lu
8178fc9973 adjust logging 2018-11-11 00:07:46 -08:00
Chris Lu
2697d6a4fb fix deleting bucket 2018-11-08 22:15:21 -08:00
Chris Lu
0d095f72e4 update to 1.04 2018-11-08 21:42:45 -08:00
Chris Lu
a4ceb051a7 use MD5 for ETag to be consistent with Amazon S3 2018-11-08 21:41:02 -08:00
Chris Lu
6e53c38c2f update to 1.03 2018-11-08 07:37:44 -08:00
Chris Lu
cbd94b18a5 improve "ls -al" performance for large directory 2018-11-08 07:37:34 -08:00
Chris Lu
6e11923551 upgrade to 1.02 2018-11-07 11:37:32 -08:00
Chris Lu
ed8efb5aef much improved "ls -al" performance 2018-11-07 11:35:13 -08:00
Chris Lu
60db731e36 update to 1.01 2018-11-05 08:54:40 -08:00
Chris Lu
f9dbaf6292 delete file by name instead of referencing the dataFile pointer 2018-11-05 08:53:38 -08:00
alex8224
8d3f9c0b75 fix invalid point when delete collection 2018-11-05 19:53:55 +08:00
Chris Lu
1a64a817ac update to 1.00 2018-11-04 22:13:16 -08:00
Chris Lu
85d1d99459 default "weed export to current enabled store" 2018-11-04 12:59:29 -08:00
Chris Lu
6c45787a37 selectively export files 2018-11-04 12:07:33 -08:00
Chris Lu
ff4189ab25 go fmt 2018-11-04 11:59:08 -08:00
Chris Lu
8dfac6a4cf working b2 sink 2018-11-04 11:58:59 -08:00
Chris Lu
f050b22d6c close file in ScanVolumeFile() 
fix https://github.com/chrislusf/seaweedfs/issues/761
 2018-11-04 00:28:24 -07:00
Chris Lu
93e0dddbf3 add function description 2018-11-03 22:32:21 -07:00
Chris Lu
a64613172d bootstrap filer replication with weed filer.export -targetStore=notification 2018-11-03 12:43:45 -07:00
Chris Lu
3674ad9f8e go fmt 2018-11-01 01:12:21 -07:00
Chris Lu
db584ff7f8 separate into notification.toml, add gcp pub/sub message queue 2018-11-01 01:11:09 -07:00
Chris Lu
6219a9ad1f filer ordered notifications 2018-10-31 21:48:05 -07:00
Chris Lu
4c97ff3717 support AWS SQS as file change notification message queue 2018-10-31 01:11:19 -07:00
Chris Lu
200cbcde62
Merge pull request #756 from zzoe/patch-1 
Update volume_server_handlers_write.go
 2018-10-30 21:52:19 -07:00
zoe
a11c28cb4c
Update volume_server_handlers_write.go 
size declared and not used
 2018-10-31 09:36:32 +08:00
Chris Lu
fe31f389b0 add configuration instructions 2018-10-30 02:29:11 -07:00
zoe
29eff0ecd7
Update volume_server_handlers_write.go 
返回实际文件大小，而不是needle的大小
 2018-10-29 21:05:51 +08:00
Chris Lu
9e98bba206 go fmt 2018-10-24 00:00:01 -07:00
Chris Lu
6a756136ef go fmt 2018-10-23 23:59:49 -07:00
Chris Lu
c58b73ad9b adding BackBlaze, Azure 2018-10-23 23:59:40 -07:00
Chris Lu
b861899919 use released version of azblob 
replace https://github.com/chrislusf/seaweedfs/pull/754
 2018-10-23 02:18:46 -07:00
Chris Lu
82a5d39778 add back volume vacuum cleanup 2018-10-18 23:02:58 -07:00
Chris Lu
496a7398d1 adjust logging 2018-10-18 20:47:30 -07:00
Chris Lu
201c24c110 add locking to volume layout vacuum 2018-10-18 20:34:43 -07:00
Chris Lu
46eb77f9bb move DiskStatus and MemStatus to protobuf 2018-10-15 22:25:28 -07:00
Chris Lu
eec951cad2 migrate volume sync to gRpc 2018-10-15 21:44:41 -07:00
Chris Lu
f8b2d3cacc move volume mount/unmount on volume server to grpc 2018-10-15 01:48:15 -07:00
Chris Lu
66a353dcb5 remove volume server /admin/volume/delete 2018-10-15 01:26:49 -07:00
Chris Lu
fda771c83f migrate volume sync status to grpc API on volume server 2018-10-15 01:19:15 -07:00
Chris Lu
b423bb9e2d migrate assign volume to grpc API on volume server 2018-10-15 00:40:46 -07:00
Chris Lu
333709657c adjust log level 2018-10-15 00:40:12 -07:00
Chris Lu
f9410bcaa7 adjust log level for deletes 2018-10-15 00:39:56 -07:00
Chris Lu
8301519fb0 migrate delete collection to grpc API on volume server 2018-10-15 00:03:55 -07:00
Chris Lu
db152ca540 fix error log format 2018-10-14 23:50:54 -07:00
Chris Lu
67d1e572ca rename files 2018-10-14 23:47:29 -07:00
Chris Lu
b1daede91b move volume vacuum to gRpc 2018-10-14 23:12:43 -07:00
Chris Lu
91ac2e0dd9 go fmt 2018-10-14 00:30:20 -07:00
Chris Lu
249eeccf32 0.99 release 2018-10-14 00:19:04 -07:00
Chris Lu
9db53cf256 use the configurable chunk size 2018-10-14 00:18:52 -07:00
Chris Lu
ff66269b62 use grpc to replace http APIs for batch volume id lookup and batch delete 
1. remove batch volume id lookup http API /vol/lookup
2. remove batch delete http API /delete
 2018-10-14 00:12:28 -07:00
Chris Lu
3ddcd87098 increase weed mount IO perforance 2018-10-14 00:10:17 -07:00
Chris Lu
ab85118233 error handling when kafka not ready 2018-10-13 23:30:00 -07:00
Chris Lu
41aba0c02c fix when no files in filer 2018-10-12 00:45:28 -07:00
Chris Lu
4af6b4eb0e fix master static resource display problem 2018-10-12 00:05:15 -07:00
Chris Lu
556382ff5f adding basic grpc to volume server 2018-10-11 01:16:33 -07:00
Chris Lu
64ed47502b rename file 2018-10-11 00:21:32 -07:00
Chris Lu
08266b7256 go fmt 2018-10-11 00:08:13 -07:00
Chris Lu
1d7e01b754 reformat 2018-10-11 00:05:54 -07:00
Chris Lu
da6154b29c refactor volume server to startVolumeServer() 2018-10-11 00:04:31 -07:00
Chris Lu
60d2f1557d refactor func name 2018-10-10 23:19:54 -07:00
Chris Lu
b7763c1cb6 releasing 0.98 2018-10-10 21:44:10 -07:00
Chris Lu
9f4c2f87d3 adding filer replication sink to Azure 2018-10-09 01:35:48 -07:00
Chris Lu
453d0be4d4 adjust error format 2018-10-09 01:35:30 -07:00
Alex Wang
4db68669b6 [bugfix] Fix interrupt hook overwritten bug 2018-10-09 15:43:32 +08:00
Chris Lu
96a23591e7 remove comments 2018-10-07 22:21:00 -07:00
Chris Lu
dbe33236fb embed static resources via statik 2018-10-07 21:42:57 -07:00
Chris Lu
fffbc2d490 embed static resources via statik 2018-10-07 10:54:05 -07:00
Chris Lu
a5b5887efc add filer replication to aws s3 2018-10-06 17:10:15 -07:00
Chris Lu
84d4a2c8a0 add filer replication to google cloud storage 2018-10-06 13:08:38 -07:00
Chris Lu
f8c2704d2b reformat 2018-10-06 13:04:33 -07:00
Chris Lu
14858f6224 add logging 2018-10-06 13:01:38 -07:00
Chris Lu
04da4c8094 add logging 2018-10-06 13:01:29 -07:00
Chris Lu
4a8ef198d7 add logging 2018-10-06 13:00:33 -07:00
Chris Lu
56c5c7b1b6 add google cloud storage 2018-10-04 01:14:44 -07:00
Chris Lu
e8ef501f02 add s3 replication sink 2018-10-03 23:36:52 -07:00
Chris Lu
56a5d5af8d ignore duplicated key error if the store has duplicated primary key prevention 
fix https://github.com/chrislusf/seaweedfs/issues/741
 2018-09-30 00:49:52 -07:00
Chris Lu
98384b62fd adjust description 2018-09-28 01:58:34 -07:00
Chris Lu
f11d531d45 adjust logging 2018-09-28 01:54:10 -07:00
Chris Lu
06c43286cd skip error when deletion failed 2018-09-28 00:10:49 -07:00
Chris Lu
746017ea4f releasing 0.97 2018-09-27 21:51:41 -07:00
Chris Lu
f7ff84f17f Revert "smalller block size for better accuracy" 
This reverts commit 97ecd20fc9.
 2018-09-27 09:42:24 -07:00
Chris Lu
97ecd20fc9 smalller block size for better accuracy 2018-09-27 09:39:17 -07:00
Chris Lu
f53632a84b add block size for linux du 
du already working on Mac
 2018-09-27 09:33:36 -07:00
Chris Lu
a362261972 avoid possible racing issue if leveldb or boltdb is used. 
remove the to-be-generated files during commit

fix https://github.com/chrislusf/seaweedfs/issues/738
 2018-09-26 18:45:51 -07:00
Chris Lu
31ed352ab6 replication handle cases when entry already exists 2018-09-25 09:27:03 -07:00
Chris Lu
60c1ada4c1 remove filer grpc API GetEntryAttributes 2018-09-25 09:18:06 -07:00
Chris Lu
578f8a6dec fix empty file reading on linux 2018-09-23 10:01:00 -07:00
Chris Lu
200d9b9ac2 add file name to log 2018-09-23 09:48:55 -07:00
Chris Lu
a60767c43a add more instructions 2018-09-23 09:26:39 -07:00
Chris Lu
12ba8b9799 generate useable filer.toml 2018-09-23 09:26:25 -07:00
Chris Lu
b1b8c4ed32 join via filepath 2018-09-23 01:46:50 -07:00
Chris Lu
b16d47030e save and use last processed kafka offsets 2018-09-23 01:34:40 -07:00
Chris Lu
9fe24991d5 refactoring 2018-09-23 00:40:36 -07:00
Chris Lu
7d6b2a4740 add ETag to upload results and chunks 2018-09-22 22:12:21 -07:00
Chris Lu
420f068376 add more instructions 2018-09-22 22:12:06 -07:00
Chris Lu
d3e1f20dcc adjust log level 2018-09-22 22:11:49 -07:00
Chris Lu
13d6a74eb2 consolidate file chunks during create 2018-09-22 22:11:37 -07:00
Chris Lu
fdc8a267de remove unused function 2018-09-22 22:11:13 -07:00
Chris Lu
9a3d46f8e3 handle duplicated replication when retry happens 2018-09-22 11:14:04 -07:00
Chris Lu
01ceace18e adjust sink options 2018-09-22 00:53:52 -07:00
Chris Lu
402ab598b6 go fmt 2018-09-22 00:12:10 -07:00
Chris Lu
7bb62b9f21 delay file creation without chunks from dir.Create to file.Flush 2018-09-22 00:11:46 -07:00
Chris Lu
db69ce89f0 go fmt 2018-09-21 01:56:43 -07:00
Chris Lu
a6cfaba018 able to sync the changes 2018-09-21 01:54:29 -07:00
Chris Lu
a8cd7df44a let filer choose default replication 2018-09-20 09:00:15 -07:00
Chris Lu
d5149e592e add logging on kafka reading 2018-09-19 22:31:45 -07:00
Chris Lu
d9b32db8d7 adjust uploads to a hidden folder 2018-09-19 22:03:16 -07:00
Chris Lu
f5471bcebf add bucket name in the redirection 2018-09-19 22:01:41 -07:00
Chris Lu
25fb6f9a46 fix compilation 2018-09-17 02:23:21 -07:00
Chris Lu
779641e9d4 adjust replicated entry name 2018-09-17 01:37:24 -07:00
Chris Lu
b7a375f5aa set large limit for "weed filer.export" directory listing 2018-09-17 00:42:36 -07:00
Chris Lu
4e1f1a0b92 set large limit for "weed mount" directory listing 2018-09-17 00:41:33 -07:00
Chris Lu
788acdf527 add WIP filer.replicate 2018-09-17 00:27:56 -07:00
Chris Lu
865a017936 fix when if buffer is not aligned 2018-09-17 00:27:40 -07:00
Chris Lu
97575e7185 do not set attributes if the file is still open 2018-09-16 12:37:06 -07:00
Chris Lu
70e67b7c6d add notification log 2018-09-16 11:20:36 -07:00
Chris Lu
5c6e3f457f adjust log 2018-09-16 11:20:08 -07:00
Chris Lu
bc03233364 refactoring 2018-09-16 01:37:35 -07:00
Chris Lu
d923ba2206 renaming msgqueue to notification 2018-09-16 01:18:30 -07:00
Chris Lu
bea4f6ca14 go fmt 2018-09-12 01:00:57 -07:00
Chris Lu
f628953c33 cleanup tmp upload directories 2018-09-12 01:00:51 -07:00
Chris Lu
f6d8525d1d working S3 multipart uploads 2018-09-12 00:46:12 -07:00
Chris Lu
439e88b5a6 fix format error 2018-09-11 13:01:51 -07:00
Chris Lu
267201ff44 handle non continuous writes 
fix #728
 2018-09-10 02:39:41 -07:00
Chris Lu
109385e955 simplifying logic to avoid handling non continuous writes 
fix https://github.com/chrislusf/seaweedfs/issues/728
 2018-09-10 02:21:57 -07:00
Chris Lu
296acc4a0a go fmt 2018-09-09 16:26:11 -07:00
Chris Lu
164091c269 add s3 multipart upload 2018-09-09 16:25:43 -07:00
Chris Lu
9b3bf0e46c fix "weed backup" rerunning 
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
 2018-09-09 02:48:58 -07:00
Chris Lu
bc8d34143d Revert "fix "weed backup" rerunning" 
This reverts commit 1937be2bb2.
 2018-09-09 02:38:30 -07:00
Chris Lu
1937be2bb2 fix "weed backup" rerunning 
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
 2018-09-09 02:31:12 -07:00
Chris Lu
d7a4b12d2a fix build error 2018-09-07 14:21:10 -07:00
Chris Lu
83ca595472 adjust version number 2018-09-07 14:19:48 -07:00
Chris Lu
747377142b fix file permission during writes 2018-09-07 13:57:06 -07:00
Chris Lu
dd37f0540f add extended attributes 2018-09-07 13:12:52 -07:00
Chris Lu
75e749039b fix data writes error when consecutive calls overlaps with previous writes 2018-09-07 13:11:43 -07:00
Chris Lu
f177cec509 avoid slice out of bounds 
avoid this problem

2018/09/04 16:27:14 fuse: panic in handler for Write [ID=0x27c0d Node=0x2 Uid=0 Gid=0 Pid=0] 0x1 131072 @10607788032 fl=WriteCache lock=0 ffl=OpenReadOnly: runtime error: slice bounds out of range
goroutine 211141 [running]:
bazil.org/fuse/fs.(*Server).serve.func2(0x10d3e60, 0xc00014be30, 0xc00052fef8, 0xc00052fe77)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:857 +0x1ac
panic(0xe2d080, 0x17f62b0)
	/home/travis/.gimme/versions/go/src/runtime/panic.go:513 +0x1b9
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).saveToStorage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0xc0005dc000, 0x20000, 0x1000000, 0x276720000, 0xc0003feaa0, 0x0, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:142 +0x8ec
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).saveExistingPagesToStorage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0x0, 0x0, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:107 +0x6c
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).AddPage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0x278460000, 0xc011966050, 0x20000, 0x20fb0, 0x6fc23ac00, 0x4a817c800, 0x0, ...)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:70 +0x8f
github.com/chrislusf/seaweedfs/weed/filesys.(*FileHandle).Write(0xc000548410, 0x10d7ba0, 0xc0003fcc00, 0xc00014be30, 0xc011946af8, 0x47fa01, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/filehandle.go:141 +0x245
bazil.org/fuse/fs.(*Server).handleRequest(0xc0002cc0c0, 0x10d7ba0, 0xc0003fcc00, 0x10cb020, 0xc000394140, 0xc0000acac0, 0x10d3e60, 0xc00014be30, 0xc00052fef8, 0x10ca6a0, ...)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:1265 +0x1599
bazil.org/fuse/fs.(*Server).serve(0xc0002cc0c0, 0x10d3e60, 0xc00014be30)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:878 +0x410
bazil.org/fuse/fs.(*Server).Serve.func1(0xc0002cc0c0, 0x10d3e60, 0xc00014be30)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:425 +0x6e
created by bazil.org/fuse/fs.(*Server).Serve
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:423 +0x321
 2018-09-05 02:17:04 -07:00
Chris Lu
d614716ff2 fix compilation error 2018-09-04 00:43:46 -07:00
Chris Lu
ce6a2140a2 preparing to support S3 multipart uploads 2018-09-04 00:42:44 -07:00
Chris Lu
7f760f16b0 prepare for 0.95 2018-09-03 20:52:52 -07:00
Chris Lu
35bafd9861 remove unused function 2018-09-03 16:50:32 -07:00
Chris Lu
7ad5a90127 go fmt 2018-09-03 16:47:00 -07:00
Chris Lu
2ed6d8cca6 refactoring filer rm 2018-09-03 13:16:26 -07:00
Chris Lu
03f852c799 refactoring 2018-09-03 13:03:16 -07:00
Chris Lu
0b0ece9649 add place holders for multiplarts upload 2018-09-03 11:38:10 -07:00
Chris Lu
0adb2ba6b8 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2018-09-03 00:29:02 -07:00
Chris Lu
06b81c063a fix possible connection counting error 2018-09-03 00:28:58 -07:00
zhangmingfeng
aafe8d84cb
Update volume_server_handlers_read.go 
change ext to lower-case
 2018-09-03 11:01:35 +08:00
Chris Lu
88f1d32cc4 add s3ChunkedReader 
fix https://github.com/chrislusf/seaweedfs/issues/718
 2018-09-02 14:20:47 -07:00
Chris Lu
76cbe8bf33 instant notification of new volumes added or deleted 2018-08-24 01:26:56 -07:00
Chris Lu
5ccf8e8078 reformat 2018-08-24 00:30:03 -07:00
Chris Lu
ac793a3c5a clean remove all left over files when deleting a collection 2018-08-23 23:33:16 -07:00
Chris Lu
d361a1ddf5 fix id String() instead 2018-08-23 22:44:30 -07:00
Chris Lu
694f93de80 parse needle id correctly 
fix https://github.com/chrislusf/seaweedfs/issues/713
 2018-08-23 20:48:43 -07:00
Chris Lu
37e4e022fd prepare for 0.94 release 2018-08-23 00:02:13 -07:00
Chris Lu
6e3f4d1079 add working filer.export command 2018-08-23 00:02:04 -07:00
Chris Lu
98b8f8649d adding filer.export prototype 2018-08-22 23:54:22 -07:00
Chris Lu
2b4277ec18 fix nil bug with filer.toml 2018-08-22 23:54:00 -07:00
Chris Lu
ae4e654d2a add example configuration for default redis create-cluster setup 2018-08-22 18:23:19 -07:00
Chris Lu
b23883b3b6 fix a bug on error reporting 2018-08-22 18:19:26 -07:00
Chris Lu
1f030af06f adjusting startup logging 2018-08-22 13:59:04 -07:00
Chris Lu
bb92fd1db0 fix test error 2018-08-19 18:52:50 -07:00
Chris Lu
53d171cf5b releasing 0.93 2018-08-19 18:45:26 -07:00
Chris Lu
906472b21d filer add upload and drag&dorp 2018-08-19 18:42:40 -07:00