mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-21 15:28:14 +08:00
Code Issues Packages Projects Releases Wiki Activity
11,193 Commits 32 Branches 296 Tags 152 MiB
7eb8257126
Commit Graph

7221 Commits

Author SHA1 Message Date
bingoohuang
6ddfaf33cb extract VolumeFileScanner for ScanVolumeFile 2019-01-16 17:48:59 +08:00
bingoohuang
27093bc2e5 fix needle Append return offset to avoid uint32 overflow 2019-01-16 16:48:03 +08:00
bingoohuang
219838db87
fail fast when failed to write local volume 2019-01-15 15:47:43 +08:00
Chris Lu
2ff95ead57
Merge pull request #832 from PapaYofen/fix-830 
Fix https://github.com/chrislusf/seaweedfs/issues/830
 2019-01-12 17:26:51 -08:00
chenwanli
b71d6f4b88 Fix https://github.com/chrislusf/seaweedfs/issues/830 2019-01-12 10:07:19 +08:00
chenwanli
80cf99db1b Set rpc dial timeout to 5s 2019-01-11 18:23:31 +08:00
Chris Lu
8613e4f558
Merge pull request #827 from suilongfei/master 
change log_dir to logdir for the "flag redefined:log_dir" error when use another package with "github.com/golang/glog"
 2019-01-10 09:10:07 -08:00
Chris Lu
2d23d86fd3 no timeout for volume vacuum 
revert changes on volume vacuum timemout from https://github.com/chrislusf/seaweedfs/pull/829
 2019-01-10 09:07:40 -08:00
chenwanli
39c7455881 Set http timeout to 5s 2019-01-10 20:21:08 +08:00
chenwanli
0a3e83a36a Set timeout for master and volume non-streaming rpc 2019-01-10 19:41:03 +08:00
隋龙飞@视频结构化
b864cf7e07
change log_dir flag to logdir 
change log_dir flag to logdir
 2019-01-10 17:17:32 +08:00
隋龙飞@视频结构化
ea4f29d1e7
change log_dir flag to logdir 
change log_dir flag to logdir
 2019-01-10 17:16:58 +08:00
Chris Lu
2a75a36b27 set cassandra ttl with entry.TTLSec 2019-01-08 20:26:19 -08:00
byungjo.yoon
58fd2e5a19 Set redis key ttl with entry.TTLSec 2019-01-09 11:48:19 +09:00
Chris Lu
badd8fdf8f shortcut for empty file 2019-01-08 09:03:28 -08:00
chenwanli
41fe31b5d0 Speed up heartbeat to new master leader 2019-01-07 16:09:15 +08:00
Chris Lu
e768e64fbc 1.20 2019-01-06 10:52:26 -08:00
Chris Lu
bd32108a90 disable async file deletion 2019-01-05 22:51:50 -08:00
Chris Lu
be9a7592a1 incrementally calculate visible intervals 2019-01-05 22:23:44 -08:00
Chris Lu
6b5d6bb5a6 adjust error names 2019-01-05 19:52:38 -08:00
Chris Lu
9383c91eb1 wait to read again if the volume is compacting 2019-01-05 19:52:17 -08:00
Chris Lu
fe50224ea0 refactoring 2019-01-05 15:21:56 -08:00
Chris Lu
6740427638 weed mount: reduce default chunk size 2019-01-05 15:17:03 -08:00
Chris Lu
d9f9aca453 refactoring 2019-01-05 15:16:39 -08:00
Chris Lu
251b596bc5 increase batch deletion size to 1024 2019-01-03 22:21:57 -08:00
Chris Lu
aac11fb832 reduce directory list cache usage 2019-01-03 22:21:39 -08:00
Chris Lu
7f74577db1 add Last-Modified to HEAD requests 
to work better with restic
 2019-01-02 14:23:25 -08:00
Chris Lu
ceafb7d145 1.19 2019-01-02 12:59:20 -08:00
Chris Lu
28a41fda3e assign new file id even on filer PUT operations 2019-01-02 12:58:26 -08:00
Chris Lu
d67f7ddfaf adjust log 2019-01-02 12:58:06 -08:00
Chris Lu
e49a38842a set etag in response 2019-01-02 12:57:54 -08:00
Chris Lu
d5ebfd5b0c fix s3cmd put 
1. fix volume put data size
 2019-01-02 12:57:33 -08:00
Chris Lu
3339325334 fix s3 ListAllMyBucketsResult to work with s3cmd 2019-01-02 11:36:29 -08:00
Chris Lu
43db7ac123 fix spelling 2019-01-02 02:38:32 -08:00
Chris Lu
9539823c35 fix log 2019-01-01 03:04:44 -08:00
Chris Lu
525db94d9a async file chunk deletion 2019-01-01 02:33:57 -08:00
Chris Lu
ffb5d3f93a get bytes from sync pool 2019-01-01 02:14:40 -08:00
Chris Lu
832ac8129a reduce pointer usage 2018-12-31 15:10:14 -08:00
Chris Lu
1478d7ea21 reduce file seek when writing 2018-12-31 15:08:32 -08:00
Chris Lu
81904ad336 debugging: track all memory allocations 2018-12-31 14:54:13 -08:00
Chris Lu
03cfb4267f adjust vaccum logging 2018-12-31 00:06:52 -08:00
Chris Lu
d6181076bb FUSE remove adjust error 2018-12-30 22:29:27 -08:00
Chris Lu
16e8610426 1.18 2018-12-30 00:52:20 -08:00
Chris Lu
0388d421d2 caching visible intervals for read 
speeds up 4x in single thread mode
speeds up 30% in 32 threads mode
 2018-12-30 00:51:44 -08:00
Chris Lu
55f7d8f801 remove extra log 2018-12-29 18:00:19 -08:00
Chris Lu
170ed53da5 weed mount: mac add AutoXattr 2018-12-29 13:55:38 -08:00
Chris Lu
d8c7844d2e 1.17 2018-12-29 13:16:38 -08:00
Chris Lu
9f3836072a weed mount: no mac finder broswe mode 2018-12-29 13:16:23 -08:00
Chris Lu
d7bf6c5b91 switching_to_seaweedfs_fuse 2018-12-29 10:31:36 -08:00
Chris Lu
0c932d1738 add volume delete grpc API 
fix https://github.com/chrislusf/seaweedfs/issues/808
 2018-12-29 00:03:30 -08:00
Chris Lu
03c7447ad6 set mount point to existing permissions 
fix https://github.com/chrislusf/seaweedfs/issues/806
 2018-12-28 23:36:13 -08:00
Chris Lu
a8af1e3f5f set attributes include crtime 2018-12-28 22:37:18 -08:00
Chris Lu
ba9b7b62d7 1.16 2018-12-28 15:10:49 -08:00
Chris Lu
79168ab0bb adjust error and log 2018-12-28 15:10:23 -08:00
Chris Lu
308ac1d0d2 reuse buffer by sync pool 2018-12-28 03:27:48 -08:00
Chris Lu
260b038441 fix url 2018-12-27 14:43:36 -08:00
Chris Lu
2a19511ba8 avoid gzip benchmark data 
related to https://github.com/chrislusf/seaweedfs/issues/804
 2018-12-27 13:10:46 -08:00
Chris Lu
d3660ffce8 avoid gzipping files without file extensions 
fix https://github.com/chrislusf/seaweedfs/issues/804
 2018-12-27 12:17:05 -08:00
Chris Lu
253f190f48 weed mount add symlink support 2018-12-25 22:45:44 -08:00
Chris Lu
9eb2bee86a avoid error log if not found 2018-12-24 17:45:26 -08:00
Chris Lu
ba923e2864 error if the last arguement does not end with forward slash 2018-12-23 18:20:11 -08:00
Chris Lu
332668a3fd 1.15 2018-12-23 12:54:41 -08:00
Chris Lu
c043fd17cb detect more gzippable content 2018-12-22 15:05:31 -08:00
Chris Lu
49f40cd145 adjust help message 2018-12-22 15:05:15 -08:00
Chris Lu
852ee21835 avoid .gz auto decompression 2018-12-22 13:58:16 -08:00
Chris Lu
be946c9e54 filer store original data size instead of data size after gzip 2018-12-22 13:11:07 -08:00
Chris Lu
141d302492 fix binarySearchCompactSection 2018-12-22 11:52:58 -08:00
Chris Lu
d3839fe279 go fmt 2018-12-22 11:11:57 -08:00
Chris Lu
5333f2984a streaming updates for large entries or large index file 
fix https://github.com/chrislusf/seaweedfs/issues/801
 2018-12-22 11:10:08 -08:00
Chris Lu
36d13355bb compact map fix bug when iterating 2018-12-22 11:05:29 -08:00
Chris Lu
f64dc7465b remove index file 
fix https://github.com/chrislusf/seaweedfs/issues/793
 2018-12-17 20:33:32 -08:00
Chris Lu
16634235c1 1.14 and HCFS 1.0.3 2018-12-17 10:03:03 -08:00
Chris Lu
52712782a2 adding link 2018-12-17 00:31:23 -08:00
Chris Lu
2fb740a2b9 fix upload path 2018-12-17 00:21:34 -08:00
Chris Lu
7b7f4215aa filer update existing entry 2018-12-17 00:20:00 -08:00
Chris Lu
6071be104b adjust log format 2018-12-17 00:03:14 -08:00
Chris Lu
3ac54792e1 paginate when filer deleting and FUSE mount renaming 2018-12-16 23:20:08 -08:00
Chris Lu
93d0ad09af error log format 2018-12-16 17:37:10 -08:00
Chris Lu
66d04cb83f 1.12 2018-12-15 08:17:24 -08:00
Chris Lu
42cb9b76c4 reduce needle map memory usage by 25% 2018-12-15 05:55:56 -08:00
Chris Lu
42b8f1145a weed mount ls paginate the results 
fix https://github.com/chrislusf/seaweedfs/issues/787
 2018-12-11 07:41:50 -08:00
Chris Lu
39ea8484c1 adjust proto API 2018-12-11 07:17:10 -08:00
Chris Lu
168f461761 fix MaxUint8 2018-12-10 20:57:51 -08:00
Chris Lu
8f78a069c6 fix possible name overflow when len(name)>255 2018-12-10 20:57:01 -08:00
Chris Lu
4a490fe35a go fmt 2018-12-09 01:27:25 -08:00
Chris Lu
5741134ca0 change overflow from map to slice 2018-12-09 01:27:11 -08:00
Chris Lu
d9189c857c remove unused file 2018-12-09 00:30:53 -08:00
Chris Lu
df4b9df142 add memory usage report 2018-12-09 00:15:23 -08:00
Chris Lu
ec5a305624 refactor 2018-12-09 00:12:37 -08:00
Chris Lu
9d3be33e5c optimize a little bit 2018-12-08 21:45:14 -08:00
Chris Lu
36d1e739e6 1.11 2018-12-08 00:32:34 -08:00
Chris Lu
1bfb96f34d optimization for reading whole chunk with gzip encoding 2018-12-07 01:57:55 -08:00
Chris Lu
29f1673d97 refactoring 2018-12-07 01:25:01 -08:00
Chris Lu
c28e8a2397 refactoring 2018-12-06 00:44:41 -08:00
Chris Lu
ffa2827ab1 fail fast if two notification queues or inputs are enabled 2018-12-06 00:37:59 -08:00
Chris Lu
63619f8b8e fail early if two filer stores are defined 2018-12-06 00:32:36 -08:00
Chris Lu
52b24a9902 default "weed server -filer" to same directory as -mdir 2018-12-05 23:24:25 -08:00
Chris Lu
fb43a9d7b9 adjust message 2018-12-05 02:03:03 -08:00
chenwanli
31038acccf fix https://github.com/chrislusf/seaweedfs/issues/780 2018-12-05 14:06:16 +08:00
Chris Lu
650d1af978 HCFS support change user name and group names 2018-12-04 00:46:00 -08:00
Chris Lu
b0838db5a8 support hdfs user name and group names 2018-12-04 00:13:40 -08:00
Chris Lu
7ace0efd65 revert prev 2018-12-02 23:25:37 -08:00
Chris Lu
738247d47e do not throw error if not found 2018-12-02 23:20:29 -08:00
Chris Lu
c6b646e92c skip creation for root 2018-12-02 23:16:17 -08:00
Chris Lu
bea162c34c add warning for empty attributes 2018-12-02 22:57:59 -08:00
Chris Lu
f0677c5af1 default root to 0755 2018-12-02 19:59:47 -08:00
Chris Lu
db22666a30 add default root directory 2018-12-02 19:42:50 -08:00
Chris Lu
a3def2bbd9 for 1.10 2018-11-29 09:15:04 -08:00
Chris Lu
7a6f49cd42 refactor a bit more 
fix https://github.com/chrislusf/seaweedfs/issues/777
 2018-11-29 00:07:54 -08:00
Chris Lu
ceca078acb avoid overwriting file or directory 
fix https://github.com/chrislusf/seaweedfs/issues/777
 2018-11-29 00:00:56 -08:00
Chris Lu
1cbd53c01c WIP SeaweedFileSystem added mkdirs, getFileStatus, listStatus, delete 2018-11-25 13:43:26 -08:00
Chris Lu
871dee4674 fix test 2018-11-23 12:37:09 -08:00
Chris Lu
86dd933596 go fmt 2018-11-23 00:26:15 -08:00
Chris Lu
444dfded84 add fs.FSStatfser for SeaweedFS weed mount 2018-11-23 00:24:51 -08:00
Chris Lu
2e32b44061 add template for fs.FSStatfser 2018-11-22 23:05:22 -08:00
Chris Lu
6344469a9b set fuse subtype 2018-11-22 23:04:23 -08:00
Chris Lu
d64546c791 adjust logs 2018-11-22 14:31:10 -08:00
Chris Lu
e12b6c0306 add subimit count parameter 
related to https://github.com/chrislusf/seaweedfs/issues/631
 2018-11-22 13:45:29 -08:00
Chris Lu
0ed816d4e9 formatting error log 2018-11-22 00:42:56 -08:00
Chris Lu
19d546938a forgot to upgrade version number 2018-11-21 23:30:59 -08:00
Chris Lu
60d3681371 default images.fix.orientation to false 2018-11-21 16:50:42 -08:00
Chris Lu
99256100a0 fix tests 2018-11-21 16:35:27 -08:00
Chris Lu
e5ad2223a1 add missing intervals 
fix https://github.com/chrislusf/seaweedfs/issues/774
 2018-11-21 16:25:13 -08:00
Chris Lu
c9b3ef58de options to control filer.export verbosity 2018-11-21 15:15:40 -08:00
Chris Lu
2f2e417b68 detailed export 2018-11-21 15:12:47 -08:00
Chris Lu
b282e34dc2 async file chunk deletion 2018-11-20 20:56:28 -08:00
Chris Lu
5065d4ab2d master add grpc API for fileid assigning 2018-11-20 11:35:45 -08:00
Chris Lu
b318167cd3 batch delete file ids 2018-11-20 11:35:08 -08:00
Chris Lu
06474e5990 1.08 2018-11-18 22:43:05 -08:00
Chris Lu
ee3fe07acf fix sort bug 2018-11-18 21:59:53 -08:00
Chris Lu
469863f7f0 merge interval now is quite faster with reusing slices 2018-11-18 21:24:58 -08:00
Chris Lu
d2924c7748 merge intervals is a bit faster 2018-11-18 20:31:39 -08:00
Chris Lu
cb5a10c6a3 adjust the tests 2018-11-18 17:05:36 -08:00
Chris Lu
f1db22d48b add benchmark test for merging intervals 2018-11-18 16:49:08 -08:00
Chris Lu
4fcfc9410f cleanup 2018-11-18 11:51:38 -08:00
Chris Lu
9655dc9ca9 simpler logic 2018-11-18 10:07:30 -08:00
Chris Lu
f8eb988347 1.07 2018-11-18 08:10:15 -08:00
Chris Lu
60421b23f0 reduce weed mount memory usage 2018-11-18 07:49:14 -08:00
Chris Lu
2929563294 1.06 2018-11-14 23:51:15 -08:00
Chris Lu
d3b7965c76 caching grpc clients 2018-11-14 23:50:46 -08:00
Chris Lu
d89774cb7f cache grpc client connections to improve performance 2018-11-14 23:31:39 -08:00
Chris Lu
2e6a3c7b16 reduce memory usage 2018-11-14 22:48:54 -08:00
Chris Lu
f224b9fe28 1.05 2018-11-13 23:54:17 -08:00
Chris Lu
11b0a9cb76 fix possible error when reusing file handles 2018-11-13 23:53:17 -08:00
Chris Lu
a62adf62ca a bit faster 2018-11-12 11:19:13 -08:00
Chris Lu
affe708961 a bit faster 2018-11-11 00:43:30 -08:00
Chris Lu
a09eaf8756 adjusting, seems there are some perf benefit 2018-11-11 00:08:22 -08:00
Chris Lu
8178fc9973 adjust logging 2018-11-11 00:07:46 -08:00
Chris Lu
2697d6a4fb fix deleting bucket 2018-11-08 22:15:21 -08:00
Chris Lu
0d095f72e4 update to 1.04 2018-11-08 21:42:45 -08:00
Chris Lu
a4ceb051a7 use MD5 for ETag to be consistent with Amazon S3 2018-11-08 21:41:02 -08:00
Chris Lu
6e53c38c2f update to 1.03 2018-11-08 07:37:44 -08:00
Chris Lu
cbd94b18a5 improve "ls -al" performance for large directory 2018-11-08 07:37:34 -08:00
Chris Lu
6e11923551 upgrade to 1.02 2018-11-07 11:37:32 -08:00
Chris Lu
ed8efb5aef much improved "ls -al" performance 2018-11-07 11:35:13 -08:00
Chris Lu
60db731e36 update to 1.01 2018-11-05 08:54:40 -08:00
Chris Lu
f9dbaf6292 delete file by name instead of referencing the dataFile pointer 2018-11-05 08:53:38 -08:00
alex8224
8d3f9c0b75 fix invalid point when delete collection 2018-11-05 19:53:55 +08:00
Chris Lu
1a64a817ac update to 1.00 2018-11-04 22:13:16 -08:00
Chris Lu
85d1d99459 default "weed export to current enabled store" 2018-11-04 12:59:29 -08:00
Chris Lu
6c45787a37 selectively export files 2018-11-04 12:07:33 -08:00
Chris Lu
ff4189ab25 go fmt 2018-11-04 11:59:08 -08:00
Chris Lu
8dfac6a4cf working b2 sink 2018-11-04 11:58:59 -08:00
Chris Lu
f050b22d6c close file in ScanVolumeFile() 
fix https://github.com/chrislusf/seaweedfs/issues/761
 2018-11-04 00:28:24 -07:00
Chris Lu
93e0dddbf3 add function description 2018-11-03 22:32:21 -07:00
Chris Lu
a64613172d bootstrap filer replication with weed filer.export -targetStore=notification 2018-11-03 12:43:45 -07:00
Chris Lu
3674ad9f8e go fmt 2018-11-01 01:12:21 -07:00
Chris Lu
db584ff7f8 separate into notification.toml, add gcp pub/sub message queue 2018-11-01 01:11:09 -07:00
Chris Lu
6219a9ad1f filer ordered notifications 2018-10-31 21:48:05 -07:00
Chris Lu
4c97ff3717 support AWS SQS as file change notification message queue 2018-10-31 01:11:19 -07:00
Chris Lu
200cbcde62
Merge pull request #756 from zzoe/patch-1 
Update volume_server_handlers_write.go
 2018-10-30 21:52:19 -07:00
zoe
a11c28cb4c
Update volume_server_handlers_write.go 
size declared and not used
 2018-10-31 09:36:32 +08:00
Chris Lu
fe31f389b0 add configuration instructions 2018-10-30 02:29:11 -07:00
zoe
29eff0ecd7
Update volume_server_handlers_write.go 
返回实际文件大小，而不是needle的大小
 2018-10-29 21:05:51 +08:00
Chris Lu
9e98bba206 go fmt 2018-10-24 00:00:01 -07:00
Chris Lu
6a756136ef go fmt 2018-10-23 23:59:49 -07:00
Chris Lu
c58b73ad9b adding BackBlaze, Azure 2018-10-23 23:59:40 -07:00
Chris Lu
b861899919 use released version of azblob 
replace https://github.com/chrislusf/seaweedfs/pull/754
 2018-10-23 02:18:46 -07:00
Chris Lu
82a5d39778 add back volume vacuum cleanup 2018-10-18 23:02:58 -07:00
Chris Lu
496a7398d1 adjust logging 2018-10-18 20:47:30 -07:00
Chris Lu
201c24c110 add locking to volume layout vacuum 2018-10-18 20:34:43 -07:00
Chris Lu
46eb77f9bb move DiskStatus and MemStatus to protobuf 2018-10-15 22:25:28 -07:00
Chris Lu
eec951cad2 migrate volume sync to gRpc 2018-10-15 21:44:41 -07:00
Chris Lu
f8b2d3cacc move volume mount/unmount on volume server to grpc 2018-10-15 01:48:15 -07:00
Chris Lu
66a353dcb5 remove volume server /admin/volume/delete 2018-10-15 01:26:49 -07:00
Chris Lu
fda771c83f migrate volume sync status to grpc API on volume server 2018-10-15 01:19:15 -07:00
Chris Lu
b423bb9e2d migrate assign volume to grpc API on volume server 2018-10-15 00:40:46 -07:00
Chris Lu
333709657c adjust log level 2018-10-15 00:40:12 -07:00
Chris Lu
f9410bcaa7 adjust log level for deletes 2018-10-15 00:39:56 -07:00
Chris Lu
8301519fb0 migrate delete collection to grpc API on volume server 2018-10-15 00:03:55 -07:00
Chris Lu
db152ca540 fix error log format 2018-10-14 23:50:54 -07:00
Chris Lu
67d1e572ca rename files 2018-10-14 23:47:29 -07:00
Chris Lu
b1daede91b move volume vacuum to gRpc 2018-10-14 23:12:43 -07:00
Chris Lu
91ac2e0dd9 go fmt 2018-10-14 00:30:20 -07:00
Chris Lu
249eeccf32 0.99 release 2018-10-14 00:19:04 -07:00
Chris Lu
9db53cf256 use the configurable chunk size 2018-10-14 00:18:52 -07:00
Chris Lu
ff66269b62 use grpc to replace http APIs for batch volume id lookup and batch delete 
1. remove batch volume id lookup http API /vol/lookup
2. remove batch delete http API /delete
 2018-10-14 00:12:28 -07:00
Chris Lu
3ddcd87098 increase weed mount IO perforance 2018-10-14 00:10:17 -07:00
Chris Lu
ab85118233 error handling when kafka not ready 2018-10-13 23:30:00 -07:00
Chris Lu
41aba0c02c fix when no files in filer 2018-10-12 00:45:28 -07:00
Chris Lu
4af6b4eb0e fix master static resource display problem 2018-10-12 00:05:15 -07:00
Chris Lu
556382ff5f adding basic grpc to volume server 2018-10-11 01:16:33 -07:00
Chris Lu
64ed47502b rename file 2018-10-11 00:21:32 -07:00
Chris Lu
08266b7256 go fmt 2018-10-11 00:08:13 -07:00
Chris Lu
1d7e01b754 reformat 2018-10-11 00:05:54 -07:00
Chris Lu
da6154b29c refactor volume server to startVolumeServer() 2018-10-11 00:04:31 -07:00
Chris Lu
60d2f1557d refactor func name 2018-10-10 23:19:54 -07:00
Chris Lu
b7763c1cb6 releasing 0.98 2018-10-10 21:44:10 -07:00
Chris Lu
9f4c2f87d3 adding filer replication sink to Azure 2018-10-09 01:35:48 -07:00
Chris Lu
453d0be4d4 adjust error format 2018-10-09 01:35:30 -07:00
Alex Wang
4db68669b6 [bugfix] Fix interrupt hook overwritten bug 2018-10-09 15:43:32 +08:00
Chris Lu
96a23591e7 remove comments 2018-10-07 22:21:00 -07:00
Chris Lu
dbe33236fb embed static resources via statik 2018-10-07 21:42:57 -07:00
Chris Lu
fffbc2d490 embed static resources via statik 2018-10-07 10:54:05 -07:00
Chris Lu
a5b5887efc add filer replication to aws s3 2018-10-06 17:10:15 -07:00
Chris Lu
84d4a2c8a0 add filer replication to google cloud storage 2018-10-06 13:08:38 -07:00
Chris Lu
f8c2704d2b reformat 2018-10-06 13:04:33 -07:00
Chris Lu
14858f6224 add logging 2018-10-06 13:01:38 -07:00
Chris Lu
04da4c8094 add logging 2018-10-06 13:01:29 -07:00
Chris Lu
4a8ef198d7 add logging 2018-10-06 13:00:33 -07:00
Chris Lu
56c5c7b1b6 add google cloud storage 2018-10-04 01:14:44 -07:00
Chris Lu
e8ef501f02 add s3 replication sink 2018-10-03 23:36:52 -07:00
Chris Lu
56a5d5af8d ignore duplicated key error if the store has duplicated primary key prevention 
fix https://github.com/chrislusf/seaweedfs/issues/741
 2018-09-30 00:49:52 -07:00
Chris Lu
98384b62fd adjust description 2018-09-28 01:58:34 -07:00
Chris Lu
f11d531d45 adjust logging 2018-09-28 01:54:10 -07:00
Chris Lu
06c43286cd skip error when deletion failed 2018-09-28 00:10:49 -07:00
Chris Lu
746017ea4f releasing 0.97 2018-09-27 21:51:41 -07:00
Chris Lu
f7ff84f17f Revert "smalller block size for better accuracy" 
This reverts commit 97ecd20fc9.
 2018-09-27 09:42:24 -07:00
Chris Lu
97ecd20fc9 smalller block size for better accuracy 2018-09-27 09:39:17 -07:00
Chris Lu
f53632a84b add block size for linux du 
du already working on Mac
 2018-09-27 09:33:36 -07:00
Chris Lu
a362261972 avoid possible racing issue if leveldb or boltdb is used. 
remove the to-be-generated files during commit

fix https://github.com/chrislusf/seaweedfs/issues/738
 2018-09-26 18:45:51 -07:00
Chris Lu
31ed352ab6 replication handle cases when entry already exists 2018-09-25 09:27:03 -07:00
Chris Lu
60c1ada4c1 remove filer grpc API GetEntryAttributes 2018-09-25 09:18:06 -07:00
Chris Lu
578f8a6dec fix empty file reading on linux 2018-09-23 10:01:00 -07:00
Chris Lu
200d9b9ac2 add file name to log 2018-09-23 09:48:55 -07:00
Chris Lu
a60767c43a add more instructions 2018-09-23 09:26:39 -07:00
Chris Lu
12ba8b9799 generate useable filer.toml 2018-09-23 09:26:25 -07:00
Chris Lu
b1b8c4ed32 join via filepath 2018-09-23 01:46:50 -07:00
Chris Lu
b16d47030e save and use last processed kafka offsets 2018-09-23 01:34:40 -07:00
Chris Lu
9fe24991d5 refactoring 2018-09-23 00:40:36 -07:00
Chris Lu
7d6b2a4740 add ETag to upload results and chunks 2018-09-22 22:12:21 -07:00
Chris Lu
420f068376 add more instructions 2018-09-22 22:12:06 -07:00
Chris Lu
d3e1f20dcc adjust log level 2018-09-22 22:11:49 -07:00
Chris Lu
13d6a74eb2 consolidate file chunks during create 2018-09-22 22:11:37 -07:00
Chris Lu
fdc8a267de remove unused function 2018-09-22 22:11:13 -07:00
Chris Lu
9a3d46f8e3 handle duplicated replication when retry happens 2018-09-22 11:14:04 -07:00
Chris Lu
01ceace18e adjust sink options 2018-09-22 00:53:52 -07:00
Chris Lu
402ab598b6 go fmt 2018-09-22 00:12:10 -07:00
Chris Lu
7bb62b9f21 delay file creation without chunks from dir.Create to file.Flush 2018-09-22 00:11:46 -07:00
Chris Lu
db69ce89f0 go fmt 2018-09-21 01:56:43 -07:00
Chris Lu
a6cfaba018 able to sync the changes 2018-09-21 01:54:29 -07:00
Chris Lu
a8cd7df44a let filer choose default replication 2018-09-20 09:00:15 -07:00
Chris Lu
d5149e592e add logging on kafka reading 2018-09-19 22:31:45 -07:00
Chris Lu
d9b32db8d7 adjust uploads to a hidden folder 2018-09-19 22:03:16 -07:00
Chris Lu
f5471bcebf add bucket name in the redirection 2018-09-19 22:01:41 -07:00
Chris Lu
25fb6f9a46 fix compilation 2018-09-17 02:23:21 -07:00
Chris Lu
779641e9d4 adjust replicated entry name 2018-09-17 01:37:24 -07:00
Chris Lu
b7a375f5aa set large limit for "weed filer.export" directory listing 2018-09-17 00:42:36 -07:00
Chris Lu
4e1f1a0b92 set large limit for "weed mount" directory listing 2018-09-17 00:41:33 -07:00
Chris Lu
788acdf527 add WIP filer.replicate 2018-09-17 00:27:56 -07:00
Chris Lu
865a017936 fix when if buffer is not aligned 2018-09-17 00:27:40 -07:00
Chris Lu
97575e7185 do not set attributes if the file is still open 2018-09-16 12:37:06 -07:00
Chris Lu
70e67b7c6d add notification log 2018-09-16 11:20:36 -07:00
Chris Lu
5c6e3f457f adjust log 2018-09-16 11:20:08 -07:00
Chris Lu
bc03233364 refactoring 2018-09-16 01:37:35 -07:00
Chris Lu
d923ba2206 renaming msgqueue to notification 2018-09-16 01:18:30 -07:00
Chris Lu
bea4f6ca14 go fmt 2018-09-12 01:00:57 -07:00
Chris Lu
f628953c33 cleanup tmp upload directories 2018-09-12 01:00:51 -07:00
Chris Lu
f6d8525d1d working S3 multipart uploads 2018-09-12 00:46:12 -07:00
Chris Lu
439e88b5a6 fix format error 2018-09-11 13:01:51 -07:00
Chris Lu
267201ff44 handle non continuous writes 
fix #728
 2018-09-10 02:39:41 -07:00
Chris Lu
109385e955 simplifying logic to avoid handling non continuous writes 
fix https://github.com/chrislusf/seaweedfs/issues/728
 2018-09-10 02:21:57 -07:00
Chris Lu
296acc4a0a go fmt 2018-09-09 16:26:11 -07:00
Chris Lu
164091c269 add s3 multipart upload 2018-09-09 16:25:43 -07:00
Chris Lu
9b3bf0e46c fix "weed backup" rerunning 
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
 2018-09-09 02:48:58 -07:00
Chris Lu
bc8d34143d Revert "fix "weed backup" rerunning" 
This reverts commit 1937be2bb2.
 2018-09-09 02:38:30 -07:00
Chris Lu
1937be2bb2 fix "weed backup" rerunning 
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
 2018-09-09 02:31:12 -07:00
Chris Lu
d7a4b12d2a fix build error 2018-09-07 14:21:10 -07:00
Chris Lu
83ca595472 adjust version number 2018-09-07 14:19:48 -07:00
Chris Lu
747377142b fix file permission during writes 2018-09-07 13:57:06 -07:00
Chris Lu
dd37f0540f add extended attributes 2018-09-07 13:12:52 -07:00
Chris Lu
75e749039b fix data writes error when consecutive calls overlaps with previous writes 2018-09-07 13:11:43 -07:00
Chris Lu
f177cec509 avoid slice out of bounds 
avoid this problem

2018/09/04 16:27:14 fuse: panic in handler for Write [ID=0x27c0d Node=0x2 Uid=0 Gid=0 Pid=0] 0x1 131072 @10607788032 fl=WriteCache lock=0 ffl=OpenReadOnly: runtime error: slice bounds out of range
goroutine 211141 [running]:
bazil.org/fuse/fs.(*Server).serve.func2(0x10d3e60, 0xc00014be30, 0xc00052fef8, 0xc00052fe77)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:857 +0x1ac
panic(0xe2d080, 0x17f62b0)
	/home/travis/.gimme/versions/go/src/runtime/panic.go:513 +0x1b9
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).saveToStorage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0xc0005dc000, 0x20000, 0x1000000, 0x276720000, 0xc0003feaa0, 0x0, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:142 +0x8ec
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).saveExistingPagesToStorage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0x0, 0x0, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:107 +0x6c
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).AddPage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0x278460000, 0xc011966050, 0x20000, 0x20fb0, 0x6fc23ac00, 0x4a817c800, 0x0, ...)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:70 +0x8f
github.com/chrislusf/seaweedfs/weed/filesys.(*FileHandle).Write(0xc000548410, 0x10d7ba0, 0xc0003fcc00, 0xc00014be30, 0xc011946af8, 0x47fa01, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/filehandle.go:141 +0x245
bazil.org/fuse/fs.(*Server).handleRequest(0xc0002cc0c0, 0x10d7ba0, 0xc0003fcc00, 0x10cb020, 0xc000394140, 0xc0000acac0, 0x10d3e60, 0xc00014be30, 0xc00052fef8, 0x10ca6a0, ...)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:1265 +0x1599
bazil.org/fuse/fs.(*Server).serve(0xc0002cc0c0, 0x10d3e60, 0xc00014be30)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:878 +0x410
bazil.org/fuse/fs.(*Server).Serve.func1(0xc0002cc0c0, 0x10d3e60, 0xc00014be30)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:425 +0x6e
created by bazil.org/fuse/fs.(*Server).Serve
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:423 +0x321
 2018-09-05 02:17:04 -07:00
Chris Lu
d614716ff2 fix compilation error 2018-09-04 00:43:46 -07:00
Chris Lu
ce6a2140a2 preparing to support S3 multipart uploads 2018-09-04 00:42:44 -07:00
Chris Lu
7f760f16b0 prepare for 0.95 2018-09-03 20:52:52 -07:00
Chris Lu
35bafd9861 remove unused function 2018-09-03 16:50:32 -07:00
Chris Lu
7ad5a90127 go fmt 2018-09-03 16:47:00 -07:00
Chris Lu
2ed6d8cca6 refactoring filer rm 2018-09-03 13:16:26 -07:00
Chris Lu
03f852c799 refactoring 2018-09-03 13:03:16 -07:00
Chris Lu
0b0ece9649 add place holders for multiplarts upload 2018-09-03 11:38:10 -07:00
Chris Lu
0adb2ba6b8 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2018-09-03 00:29:02 -07:00
Chris Lu
06b81c063a fix possible connection counting error 2018-09-03 00:28:58 -07:00
zhangmingfeng
aafe8d84cb
Update volume_server_handlers_read.go 
change ext to lower-case
 2018-09-03 11:01:35 +08:00
Chris Lu
88f1d32cc4 add s3ChunkedReader 
fix https://github.com/chrislusf/seaweedfs/issues/718
 2018-09-02 14:20:47 -07:00
Chris Lu
76cbe8bf33 instant notification of new volumes added or deleted 2018-08-24 01:26:56 -07:00
Chris Lu
5ccf8e8078 reformat 2018-08-24 00:30:03 -07:00
Chris Lu
ac793a3c5a clean remove all left over files when deleting a collection 2018-08-23 23:33:16 -07:00
Chris Lu
d361a1ddf5 fix id String() instead 2018-08-23 22:44:30 -07:00
Chris Lu
694f93de80 parse needle id correctly 
fix https://github.com/chrislusf/seaweedfs/issues/713
 2018-08-23 20:48:43 -07:00
Chris Lu
37e4e022fd prepare for 0.94 release 2018-08-23 00:02:13 -07:00
Chris Lu
6e3f4d1079 add working filer.export command 2018-08-23 00:02:04 -07:00
Chris Lu
98b8f8649d adding filer.export prototype 2018-08-22 23:54:22 -07:00
Chris Lu
2b4277ec18 fix nil bug with filer.toml 2018-08-22 23:54:00 -07:00
Chris Lu
ae4e654d2a add example configuration for default redis create-cluster setup 2018-08-22 18:23:19 -07:00
Chris Lu
b23883b3b6 fix a bug on error reporting 2018-08-22 18:19:26 -07:00
Chris Lu
1f030af06f adjusting startup logging 2018-08-22 13:59:04 -07:00
Chris Lu
bb92fd1db0 fix test error 2018-08-19 18:52:50 -07:00
Chris Lu
53d171cf5b releasing 0.93 2018-08-19 18:45:26 -07:00
Chris Lu
906472b21d filer add upload and drag&dorp 2018-08-19 18:42:40 -07:00
Chris Lu
f22d6b87e1 go fmt 2018-08-19 15:42:50 -07:00
Chris Lu
748870b03f add "weed scaffold" command 2018-08-19 15:36:30 -07:00
Chris Lu
95b997fb35 go fmt 2018-08-19 15:18:37 -07:00
Chris Lu
f827ada811 merge notification config with filer.toml 2018-08-19 15:17:55 -07:00
Chris Lu
c91372daa6 add one example filer.toml 2018-08-19 14:58:24 -07:00
Chris Lu
ee920d4f83 kafka set the right topic 2018-08-19 01:27:30 -07:00
Chris Lu
708acee502 add support for redis cluster 
fix https://github.com/chrislusf/seaweedfs/issues/705
 2018-08-15 00:01:38 -07:00
Chris Lu
9b51ed7af2 fix test 2018-08-13 01:38:18 -07:00
Chris Lu
ae9b81c18f fix tests 2018-08-13 01:33:21 -07:00
Chris Lu
5c467083e5 go fmt 2018-08-13 01:22:32 -07:00
Chris Lu
f036ef8a3c add filer notification 2018-08-13 01:20:49 -07:00
Chris Lu
75d63db60d randomize raft server startup 
also some go fmt
 2018-08-12 14:27:14 -07:00
Chris Lu
eca4b928d2 ensure master server count is odd 2018-08-12 14:25:31 -07:00
Chris Lu
b931ced0a3 readable file size 2018-08-10 23:47:31 -07:00
Chris Lu
2e427c66ac small refactoring 2018-08-10 23:47:17 -07:00
Chris Lu
c81f1cda47 correctly recursively delete folders 2018-08-01 01:26:41 -07:00
Chris Lu
495a776671 support 128 bit NeedleId 2018-07-31 23:25:26 -07:00
Chris Lu
44d8e35988 add listener for volumd id location for benchmark tool 2018-07-31 19:12:36 -07:00
Chris Lu
d09606c65b refactoring 2018-07-29 02:25:24 -07:00
Chris Lu
032f1226db fix compilation error 2018-07-29 02:25:03 -07:00
Chris Lu
d3205a0070 go fmt 2018-07-28 21:02:56 -07:00
Chris Lu
7214a8e265 fix init error 2018-07-28 18:40:31 -07:00
Chris Lu
cfbfc7cb67 fix compilation error 2018-07-28 18:34:15 -07:00
Chris Lu
4e0522a80c adjust API 2018-07-28 18:17:31 -07:00
Chris Lu
888eb2abb5 filer read write all via locations from MasterClient 2018-07-28 14:51:36 -07:00
Chris Lu
1d779389cb MasterClient replicates all vid locations 2018-07-28 14:22:46 -07:00
Chris Lu
01bcc89803 refactor into MasterClient 2018-07-28 02:10:32 -07:00
Chris Lu
1ab8232b55 filer only talk to the master leader 2018-07-28 01:30:03 -07:00
Chris Lu
e8d4be579d send initial list of volume location 2018-07-28 01:17:35 -07:00
Chris Lu
a12c7b86b0 broadcast messages of new and deleted volumes 2018-07-27 23:09:55 -07:00
Chris Lu
f82ac793b4 Merge branch 'master' into add_topo_listener 2018-07-27 02:26:43 -07:00
Chris Lu
10bb76fe6b go fmt 2018-07-27 02:10:42 -07:00
Chris Lu
4d322df95e avoid reparsing the multipart form 2018-07-27 02:10:10 -07:00
Chris Lu
0d98949199 tmp commit 2018-07-27 01:54:45 -07:00
Chris Lu
e4b7e31902 add back recursively delete a folder 2018-07-24 22:33:26 -07:00
Chris Lu
57ded4fd80 speed up leveldb bolddb loading 2018-07-24 22:17:56 -07:00
Chris Lu
ea8ecf35f4 set needle AppendAtNs value during writes 2018-07-24 02:44:33 -07:00
Chris Lu
655b449270 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2018-07-24 02:20:02 -07:00
Chris Lu
0fb4f59fa6
Merge pull request #697 from chrislusf/add_version3 
Add version3
 2018-07-24 02:19:06 -07:00
Chris Lu
69d1b88b9e starting on 0.93 beta 2018-07-24 02:18:39 -07:00
Chris Lu
e8a5ee4f48 releasing 0.92 
ensure the nil problem is resolved.
 2018-07-24 01:54:15 -07:00
Chris Lu
77fadab560 go fmt 2018-07-24 01:38:08 -07:00
Chris Lu
4f317c7e3d working version3 2018-07-24 01:36:04 -07:00
Chris Lu
0b99557c29 fix compilation error 2018-07-24 01:09:32 -07:00
Chris Lu
9bf93e698e refactoring: avoid common code 2018-07-24 00:58:52 -07:00
Chris Lu
aae77e8ea3 initialize read seeker 2018-07-24 00:54:15 -07:00
Chris Lu
5a237af973 fix nil if not images 2018-07-24 00:49:53 -07:00
Chris Lu
a9069520ae start 0.92 beta2 2018-07-23 01:59:23 -07:00
Chris Lu
04356ad141 re-releasing 0.91 2018-07-23 01:58:13 -07:00
Chris Lu
6039b25bc9 s3API working with listing objects 2018-07-23 01:55:26 -07:00
Chris Lu
3c7ffa2a2c go fmt 2018-07-23 01:16:17 -07:00
Chris Lu
ebad3a44ab s3api do not proxy directory requests 2018-07-23 01:15:59 -07:00
Chris Lu
7f32eb1e25 s3 add https support 2018-07-22 21:28:54 -07:00
Chris Lu
c133e72f62 print out unsupported methods 2018-07-22 19:55:53 -07:00
Chris Lu
ff331178aa
Merge pull request #695 from zhangmingfeng/master 
增加chunk图片文件支持width和height
 2018-07-22 19:54:04 -07:00
Chris Lu
7e76a7d21b
Merge pull request #694 from himyxia/inprove_switch_case 
better way to handler switch case
 2018-07-22 10:29:37 -07:00
Jimmy
1c77d323b0 better way to handler switch case 2018-07-23 01:27:10 +08:00
Chris Lu
5e55f0da4d add WIP 0.92 beta 2018-07-22 02:15:24 -07:00
Chris Lu
19c537b687 releasing 0.91 2018-07-22 02:12:02 -07:00
Chris Lu
20853972b2 s3 API add ListObjectsV2Handler 2018-07-22 02:04:07 -07:00
Chris Lu
056871242b remove println 2018-07-22 01:17:49 -07:00
Chris Lu
0655151b8c go fmt 2018-07-22 01:15:11 -07:00
Chris Lu
6319d84f42 s3 API add ListObjectsV1 2018-07-22 01:14:36 -07:00
Chris Lu
922c614bde fix compilation problem 2018-07-21 20:42:14 -07:00
Chris Lu
16c66e097b adjust default buckets folder 2018-07-21 20:29:33 -07:00
Chris Lu
49375d6031 adjust DELETE response to S3 format 2018-07-21 19:12:44 -07:00
Chris Lu
5068b6ae7d fix compilation 2018-07-21 19:06:09 -07:00
Chris Lu
12825938d7 unused method 2018-07-21 18:50:56 -07:00
Chris Lu
8f240276db s3 add HEAD DELETE 2018-07-21 18:49:47 -07:00
Chris Lu
bfa87b4f50 adjust log level 2018-07-21 18:47:23 -07:00
Chris Lu
648626115a fix printing 2018-07-21 18:47:07 -07:00
Chris Lu
12e6ccc581 remove unused code 2018-07-21 17:54:14 -07:00
Chris Lu
702d7ac424 delegating old chunk purging to CreateEntry 
1. filer add file user id and group id
2. auto-chunking set the file name correctly
3. delegating old chunk purging to CreateEntry
 2018-07-21 17:47:59 -07:00
Chris Lu
852af28f91 needle priting format 2018-07-21 17:41:21 -07:00
Chris Lu
6f30a78a6c update also delete old chunks if different from latest chunks 2018-07-21 17:40:00 -07:00
Chris Lu
7e2031b18f go fmt 2018-07-21 17:39:10 -07:00
Chris Lu
dc8dd8b486 Merge branch 'master' into add_s3 2018-07-21 15:59:18 -07:00
Chris Lu
c98df05ed0 support PUT 2018-07-21 15:58:48 -07:00
Chris Lu
feb8eeb830 refactor: separate multipart parsing file 2018-07-21 12:01:35 -07:00
Chris Lu
8480008a9a add s3 upload, and removing mono and multi part upload analyzer 
removing mono and multi part upload analyzer, which were used just to determine the file name
 2018-07-21 10:39:02 -07:00
Chris Lu
80d80daf64 set filer2.ErrNotFound for not found entry 2018-07-20 02:14:18 -07:00
Chris Lu
dad733086a fix string printing 2018-07-20 00:58:10 -07:00
Chris Lu
e73b859b1d fix compilation error 2018-07-20 00:46:02 -07:00
Chris Lu
568132c0c9 delete bucket also removes collection data 2018-07-20 00:10:01 -07:00
Chris Lu
13e5541e17 FUSE can change file or folder attributes 
FUSE can change file or folder attributes
 2018-07-19 02:17:36 -07:00
Chris Lu
a09ef6002a adjust name to avoid confusion 2018-07-19 01:46:18 -07:00
Chris Lu
feea33d5e0 add HeadBucket 2018-07-19 01:43:27 -07:00
Chris Lu
d429b6296b fix test 2018-07-19 01:27:45 -07:00
Chris Lu
c7228fefa7 add bucket creation and deletion 
1. option for "weed s3 -filer.dir.buckets" to choose a folder for buckets
2. create a bucket
3. delete a bucket, recursively delete all metadata on filer
 2018-07-19 01:21:44 -07:00
Chris Lu
702fbdf731 refactoring 2018-07-18 02:39:12 -07:00
Chris Lu
834a25f084 add list all my buckets 2018-07-18 02:37:09 -07:00
Chris Lu
7abfab8e77 add feature to mount a specific filer path to local directory 2018-07-14 21:09:21 -07:00
Chris Lu
3edfe1d28f extend export command to show tombstone + change output format to CSV 
merging https://github.com/chrislusf/seaweedfs/pull/610 and add "-limit" option
 2018-07-14 20:51:17 -07:00
Chris Lu
85eae57e4c Store and replicate timestamp of file deletion in tombstone marker 
merging changes from https://github.com/chrislusf/seaweedfs/pull/609/files
 2018-07-14 20:26:45 -07:00
Chris Lu
67763fb375 fix data center setting error 2018-07-14 14:09:45 -07:00
Chris Lu
6d58916cfd fix compilation error 2018-07-14 13:39:57 -07:00
Chris Lu
842dab07b4 weed mount can request to prioritize to write to a data center 2018-07-14 13:36:28 -07:00
Chris Lu
a51aa4c586 avoid extra log messages 
fix https://github.com/chrislusf/seaweedfs/issues/689
 2018-07-13 22:07:32 -07:00
Chris Lu
9e974f6a1b
Merge pull request #686 from qszxnp/filer2_dir_cache_bugfix 
bugfix:delete Directory cache when filer2 delete Directory
 2018-07-12 01:17:32 -07:00
聂鹏
1ed41f8b21 bugfix:check Directory attr before create it 2018-07-12 00:53:31 -04:00
聂鹏
c523bc4f07 bugfix:delete Directory cache when filer2 delete Directory 2018-07-11 21:27:25 -04:00
Chris Lu
2fd48df9d9 fix test 2018-07-11 12:52:48 -07:00
Chris Lu
f7afa80250 also remove volumeLayout if collection is empty 2018-07-11 02:01:33 -07:00
Chris Lu
1e1c368ae8 adjust filer help message 2018-07-10 00:22:11 -07:00
Chris Lu
007c4a1965 alt request for dc preferred requests 2018-07-10 00:20:50 -07:00
Chris Lu
8a12fd9ec4 add error checking when loading dat file 2018-07-09 23:31:25 -07:00
Chris Lu
6c4fee85e3 add primary request and backup request when requesting volumes for a data center 2018-07-09 23:18:20 -07:00
Chris Lu
50d351944f filer add data center option 2018-07-09 02:22:48 -07:00
Chris Lu
d0982cafa7 refactoring 2018-07-09 00:22:50 -07:00
Chris Lu
5b1fd374be fix all tests 2018-07-08 02:39:04 -07:00
Chris Lu
d4d7ced922 refactoring: add type for needle id, offset 
later the type size can possibly be adjusted
 2018-07-08 02:28:04 -07:00
Chris Lu
922032b9bb add filer sub dir list size limit to "weed server" 2018-07-08 02:11:36 -07:00
Chris Lu
702ff48cda add dir list limit option 2018-07-07 02:18:47 -07:00
Chris Lu
2ad45ca04f refactor 2018-07-07 02:03:25 -07:00
Chris Lu
5bfb72d058 faster loading boltdb or leveldb needle map metrics by bloomfilter 
avoid btree
 2018-07-07 00:51:17 -07:00
Chris Lu
aba1fe01b3 use constant NeedleIndexSize instead of the number 16 2018-07-07 00:50:14 -07:00
Chris Lu
defcd4edc5 adjust log level 2018-07-07 00:48:58 -07:00
Chris Lu
c8234a5af6 ensure using none nil attributes 
fix https://github.com/chrislusf/seaweedfs/issues/674
 2018-07-05 23:16:34 -07:00
zhangmingfeng
79d18c69b4 增加chunk图片文件支持width和height 2018-07-05 10:34:17 +08:00
Chris Lu
77fc8c5914 keep alive for gRpc calls 2018-07-03 19:07:55 -07:00
Chris Lu
28e5f20c8e add super block extra for erasure coding 2018-06-28 22:20:37 -07:00
Chris Lu
9e6ad558c5 upgrading dependencies to latest versions 2018-06-28 22:03:47 -07:00
Chris Lu
9bcaa65574 avoid divided by 0 
fix https://github.com/chrislusf/seaweedfs/issues/650
 2018-06-25 01:20:15 -07:00
Chris Lu
0d83c1b91e refactoring 2018-06-25 00:01:53 -07:00
Chris Lu
018a9a20be add test for volume registeration at master 2018-06-24 23:55:27 -07:00
Chris Lu
cb19a7da99 add a note 2018-06-24 23:55:06 -07:00
Chris Lu
ef98d26c9e ensuring removing registered datanode when disconnecting 2018-06-24 23:20:27 -07:00
Chris Lu
afc1532b24 file rename 2018-06-24 23:19:56 -07:00
Chris Lu
233a9a9b6c fix log error 2018-06-24 16:12:41 -07:00
Chris Lu
49f4216280 fix compilation error 2018-06-24 15:37:54 -07:00
Chris Lu
7f6500ae07 super block reserves a couple of bits for future extension 2018-06-24 15:23:03 -07:00
Chris Lu
f339e9c284 add support for extra super block info 2018-06-24 15:19:57 -07:00
Chris Lu
9ba335a7c4 prepare for flexible super block 2018-06-24 11:37:08 -07:00
Chris Lu
c0122eeef3 start 0.91 beta 2018-06-24 09:29:34 -07:00
Chris Lu
cf76711789 releasing 0.90 version 2018-06-23 18:36:27 -07:00
Chris Lu
3a6f9c9bcf increase test size 2018-06-23 18:30:13 -07:00
Chris Lu
7f7e4e9885 fix error for deleted files during compaction 
deletion during commit may cause trouble when make up the difference during commitCompact()
 2018-06-23 18:24:59 -07:00
Chris Lu
3699996985 add compaction test 2018-06-23 16:48:19 -07:00
Chris Lu
14c6f12b0c adjust interface parameter name and doc 2018-06-17 13:24:57 -07:00
Chris Lu
db0c25b3d8 delete old entry only after successful insert 2018-06-17 13:02:22 -07:00
Chris Lu
eb2acd11c2 decouple from viper for filer store 2018-06-17 13:01:57 -07:00
Chris Lu
0467195f07 fix file closing on windows 2018-06-13 00:36:51 -07:00
Chris Lu
4433b4c006 treat 0 sized file as normal 
fix https://github.com/chrislusf/seaweedfs/issues/409
 2018-06-12 21:11:26 -07:00
Chris Lu
03f50180f3 simplifying the leader election by raft 
fixing https://github.com/chrislusf/seaweedfs/issues/629
 2018-06-12 01:54:09 -07:00
Chris Lu
69b4f93830 fix compilation for go tip 2018-06-12 00:58:43 -07:00
Chris Lu
17918f0b82 fix compilation error on go tip 2018-06-11 23:51:36 -07:00
Chris Lu
eaf000eff1 fix error on go tip 2018-06-11 23:39:42 -07:00
Chris Lu
d2a811eef2 fix compilation 2018-06-11 23:29:09 -07:00
Chris Lu
5bd72696ac weed mount add ttl option 2018-06-11 23:13:33 -07:00
Chris Lu
98110c1697 add collection, replication, ttl to each file 2018-06-10 16:57:32 -07:00
Chris Lu
f1273073fc switch to fs.NodeRequestLookuper 
in order to set the entry valid duration
 2018-06-07 00:07:37 -07:00
Chris Lu
cec1d97035 recursively move files and directories 2018-06-06 23:39:30 -07:00
Chris Lu
ae23e46313 better error log 2018-06-06 23:06:19 -07:00
Chris Lu
e755540be9 fix caching during directory listing 2018-06-06 22:55:59 -07:00
Chris Lu
daabdfe357 remove nodemap, fix directory listing cache 2018-06-06 22:48:51 -07:00
Chris Lu
5c25d29272 support renaming files 2018-06-06 22:11:01 -07:00
Chris Lu
b3447f4375 adjust logging 2018-06-06 02:21:36 -07:00
Chris Lu
6816661b0f fixed file handle by file full path 2018-06-06 02:09:57 -07:00
Chris Lu
299312c805 use separate filer grpc port 2018-06-05 23:37:41 -07:00
Chris Lu
95fe745a0c compatible with previous json rendering 2018-06-04 19:28:59 -07:00
Chris Lu
ebf58709d3 remove unused register file function 2018-06-01 23:24:34 -07:00
Chris Lu
43e3f5724c use fixed list of masters in both filer and volume servers 2018-06-01 00:39:39 -07:00
Chris Lu
a6f7f9b0b8 add missing modify and create times 2018-05-31 22:50:19 -07:00
Chris Lu
7efeb146c5 fix log 2018-05-31 22:49:55 -07:00
Chris Lu
4d1eedfa28 prepare to merge with master 2018-05-30 22:29:18 -07:00
Chris Lu
dc13e10637 fix copy error on 0 size files 2018-05-30 22:28:14 -07:00
Chris Lu
a218eaf1f0 fix log 2018-05-30 22:09:24 -07:00
Chris Lu
430eb67489 handle large file copy when write request is larger than buffer 2018-05-30 22:02:21 -07:00
Chris Lu
44acf4b756 fix file name 
f.Name() is not the base file name
 2018-05-30 20:48:13 -07:00
Chris Lu
0301504184 add mime, use simple insert and update filer store API 
1. add mime type to file in filer
2. purge old chunks if overwrite during insert
 2018-05-30 20:24:57 -07:00
Chris Lu
26e7cd8c75 assign a different volume on large file copying 2018-05-30 01:05:26 -07:00
Chris Lu
2fe0d479f1 filer.copy supports chunking 2018-05-30 00:54:56 -07:00
Chris Lu
2d7dea1a09 a little refactoring 2018-05-29 23:52:27 -07:00
Chris Lu
8a48baa056 add single chunk file copying to new filer 2018-05-29 23:46:45 -07:00
Chris Lu
78aabf66ce add error details 2018-05-29 23:46:17 -07:00
Chris Lu
4e3ea49cff properly working local write buffer 2018-05-29 01:21:21 -07:00
Chris Lu
c4b92e17d0 fix isPerfectAppend 2018-05-28 22:45:52 -07:00
Chris Lu
74332e1a61 minor 2018-05-28 14:32:16 -07:00
Chris Lu
be0e88a606 fix chunk size limit for default 0 2018-05-28 13:44:27 -07:00
Chris Lu
5c4480ec6c add mountOptions.chunkSizeLimitMB, remove cmdMount.IsDebug 2018-05-28 13:42:25 -07:00
Chris Lu
8ab7dd9d08 weed mount add options for collection and replication 2018-05-28 13:24:48 -07:00
Chris Lu
d0b238d2db cache local writes before flushing to volume server 2018-05-28 12:30:17 -07:00
Chris Lu
07e0d13d2d filer support reading multiple chunks, with range support 2018-05-28 05:39:12 -07:00
Chris Lu
dd5661b297 adding favicon to filer also 2018-05-28 02:35:58 -07:00
Chris Lu
9b603f5ffa split filer read into 2 files 2018-05-28 02:24:14 -07:00
Chris Lu
8db9319a06 refactoring: go fmt, reorg 2018-05-27 23:59:49 -07:00
Chris Lu
480a073f1f refactoring: split into 4 files 2018-05-27 23:53:10 -07:00
Chris Lu
5b844d7011 pass reference of master server 
SendHeartbeat passes lock by value: weed_server.MasterServer contains
sync.Mutex
 2018-05-27 11:58:00 -07:00
Chris Lu
8b0718ac92 go vet 2018-05-27 11:56:49 -07:00
Chris Lu
458ada173e go fmt 2018-05-27 11:52:26 -07:00
Chris Lu
8647191bee removing filer package! 2018-05-27 11:52:15 -07:00
Chris Lu
03d76479a5 add reds 2018-05-27 11:14:29 -07:00
Chris Lu
a6a6b87f1b adjust readme 2018-05-27 11:14:12 -07:00
Chris Lu
ab86d263c9 add Cassandra 2018-05-27 00:01:15 -07:00
Chris Lu
603de2d5db skip deletion if entry not found 2018-05-27 00:00:56 -07:00
Chris Lu
f124ebab1d fix ordering 2018-05-27 00:00:11 -07:00
Chris Lu
1c0a2a7db8 reformat 2018-05-26 23:59:56 -07:00
Chris Lu
869161a261 support both mysql and postgres 2018-05-26 22:02:49 -07:00
Chris Lu
87b3b84471 simplifying dirhash to 64bit integer 2018-05-26 21:24:03 -07:00
Chris Lu
955eae3500 this works great and support long file names 2018-05-26 14:08:55 -07:00
Chris Lu
2da84ed331 working fine now 
but index length can be improved
 2018-05-26 13:35:56 -07:00
Chris Lu
68bcaff14d mysql can compile, not yet tested! 2018-05-26 05:32:15 -07:00
Chris Lu
c1e353491a refactoring variable names 2018-05-26 04:50:55 -07:00
Chris Lu
828e4a5ace fix test error 2018-05-26 04:13:50 -07:00
Chris Lu
9e77563c99 add leveldb store 
1. switch to viper for filer store configuration
2. simplify FindEntry() return values, removing “found”
3. add leveldb store
 2018-05-26 03:49:46 -07:00
Chris Lu
c34feca59c refactoring 2018-05-25 23:27:06 -07:00
Chris Lu
6de84c64c6 adding create time 2018-05-25 23:26:40 -07:00
Chris Lu
72fc001d0d remove unused code 2018-05-25 01:30:16 -07:00
Chris Lu
ac66c133a5 do not read attributes when file is opened 2018-05-25 01:27:21 -07:00
Chris Lu
6d1bcd4b8c use existing attributes instead of fetching from filer 2018-05-25 01:22:31 -07:00
Chris Lu
0a223838bd refactoring 2018-05-25 00:57:25 -07:00
Chris Lu
f8776ad5cd add breadcrumb to filer UI 2018-05-24 23:42:13 -07:00
Chris Lu
a01557f4cb adjust filer web page 2018-05-24 23:21:12 -07:00
Chris Lu
c11d84f314 fix reading from a url 2018-05-24 23:20:56 -07:00
Chris Lu
7b81cf3762 better logs 2018-05-24 23:20:26 -07:00
Chris Lu
9f4f8de9ad skip printout 2018-05-24 23:20:12 -07:00
Chris Lu
e18c7e160c avoid empty chunk view 2018-05-24 23:19:56 -07:00
Chris Lu
d773e11c7a file handler directly read from volume servers 
this mostly works fine now!

next: need to cache files to local disk
 2018-05-24 01:22:37 -07:00
Chris Lu
00d0274fd7 prepare to read from multiple file chunks 2018-05-23 22:28:54 -07:00
Chris Lu
849b6ec28d seems editing already working 
Need to handle multiple chunks read.
Need to cache local file changes.
 2018-05-23 20:55:24 -07:00
Chris Lu
d84b80b795 similar changes as master 2018-05-23 03:23:47 -07:00
Chris Lu
536559f62d copy works, edit somehow still fails 2018-05-23 03:08:46 -07:00
Chris Lu
1675243f29 maybe speed up a little when loading index 2018-05-23 02:38:11 -07:00
Chris Lu
e97c60cc10 avoid overwriting variables 2018-05-23 02:36:19 -07:00
Chris Lu
468514f525 reduce logs 2018-05-23 02:36:06 -07:00
Chris Lu
69b9d8c3c2 fix boltdb variable usage 2018-05-22 10:18:09 -07:00
Chris Lu
873868cc10 not working now 
need to add file handler
 2018-05-22 04:31:44 -07:00
Chris Lu
7362de9a18 weed mount can work well 
TODO: somehow filer url is returning empty content
 2018-05-22 03:26:38 -07:00
Chris Lu
9dd228747c filer copy added uid/gid 2018-05-21 01:25:30 -07:00
Chris Lu
f07482382b able to update file content 
having some issue when vi reports file changed.
 2018-05-21 00:00:28 -07:00
Chris Lu
cd47528a75 remove unused code 2018-05-20 17:08:54 -07:00
Chris Lu
cc66e25cd2 merge intervals 2018-05-20 17:06:09 -07:00
Chris Lu
7ca5052942 create files correctly! 2018-05-19 13:51:44 -07:00
Chris Lu
793dd81ca2 skip permission checking when creating dir or files 2018-05-19 12:40:24 -07:00
Chris Lu
ab4ddb1e0e fix directory creation, directory listing 2018-05-19 12:07:15 -07:00
Chris Lu
e31c514b00 adding modified time to file chunk 2018-05-16 00:54:44 -07:00
Chris Lu
6bf31467c7 adding empty fsync 
less error now. But still:

vi on write: E514: write error (file system full?)
cp: Input/output error
 2018-05-16 00:54:27 -07:00
Chris Lu
b303a02461 cp file can work 
1. consolidate to filer_pb.FileChunk
2. dir add file, mkdir
3. file flush, write

updates having issue
 2018-05-16 00:08:44 -07:00
Chris Lu
c7a71d35b0 fix on pagination 2018-05-14 20:27:48 -07:00
Chris Lu
58954bf46f pass file attributes from filer to mount 2018-05-14 02:02:17 -07:00
Chris Lu
67401f1908 fix gRpc nil response 2018-05-14 00:17:22 -07:00
Chris Lu
c5cf9bd290 properly working filer 2018-05-13 23:56:16 -07:00
Chris Lu
f01d5616b3 add better listing directory entries 2018-05-13 14:02:29 -07:00
Chris Lu
a4740ca836 fix compilation error for windows 2018-05-13 00:49:59 -07:00
Chris Lu
2dd5b777f7 prepare for 0.77 release 
right before changing to filer2
 2018-05-13 00:26:23 -07:00
Chris Lu
75a2b05aa3 remove Size from Attr 2018-05-13 00:11:26 -07:00
Chris Lu
fc7f81f6e7 remove nlink property from directory node 2018-05-12 22:42:28 -07:00
Chris Lu
9ecbf92569 more tests 2018-05-12 13:50:11 -07:00
Chris Lu
81af1bafba file store adds memdb implementation 2018-05-12 13:45:29 -07:00
Chris Lu
a808df5019 visit parent folder first 2018-05-11 02:27:57 -07:00
Chris Lu
8a1d640dc4 starting filer2 2018-05-11 02:20:15 -07:00
Chris Lu
9f345da20f mv filer proto to filer_pb 2018-05-09 23:18:02 -07:00
Chris Lu
942c2cbd7b mv pb to master_pb 2018-05-09 23:11:54 -07:00
Chris Lu
57c5405657 add todo 2018-05-09 04:18:31 -07:00
Chris Lu
6db1c8bafb Revert "adjust the needle mapper" 
This reverts commit e2c7be3a2e.
 2018-05-09 03:57:07 -07:00
Chris Lu
e2c7be3a2e adjust the needle mapper 
as discussed on
https://github.com/chrislusf/seaweedfs/pull/642#issuecomment-387606960
 2018-05-09 03:50:14 -07:00
Chris Lu
43a69d20bf change filer API to gRPC 2018-05-08 01:59:43 -07:00
Chris Lu
4936d6c342 adjusting for FUSE 2018-05-07 00:37:47 -07:00
Chris Lu
cb6b1d8d41 fix broken master health checking 2018-05-07 00:37:33 -07:00
Chris Lu
be58993f47 ensure the same Node is returned 2018-05-05 23:50:34 -07:00
Chris Lu
b857cf9d9b format 2018-05-05 23:39:29 -07:00
Chris Lu
456738ba64 refactoring fuse 2018-05-05 22:47:16 -07:00
Chris Lu
2cada598c6 fix test 2018-05-05 15:08:23 -07:00
Chris Lu
f3e07c231b adding folder icon 2018-05-05 14:41:40 -07:00
Chris Lu
7f49514f6f fix filer UI 2018-05-05 14:10:50 -07:00
Chris Lu
fffb14bc87 better support FUSE Lookup() 2018-05-05 02:01:50 -07:00
Chris Lu
050ab19264 add design 2018-05-04 23:38:42 -07:00
Chris Lu
94a35f25f3 simple refactoring 2018-05-03 00:16:02 -07:00
WenLong LI
b9068c1304
use volumes variable 
fix leveldb mount bug
 2018-05-03 12:13:53 +08:00
liwenlong05
02a1597642 fix leveldb mount bug 2018-05-03 11:20:38 +08:00
mtolman
e4b1e5c3c8
Fix for issue #635 
Fix for the following issue on master: https://github.com/chrislusf/seaweedfs/issues/635
 2018-04-17 13:26:48 -06:00
Chris Lu
d49e91cd78 cassandra use local quorum for better performance 2018-04-16 01:33:05 -07:00
Chris Lu
15b8499f8d refactoring 2018-04-16 01:16:06 -07:00
Chris Lu
d712c87ec0 update glide 2018-04-16 00:59:10 -07:00
Chris Lu
6448e0c25d update glide 2018-04-16 00:38:49 -07:00
Chris Lu
8c64d06b3a update redis library 2018-04-16 00:27:35 -07:00
Chris Lu
9cd6b33e97 fix https://github.com/chrislusf/seaweedfs/issues/603 2018-02-27 14:18:00 -08:00
Chris Lu
4b9b9547d4 unused option 2018-02-27 13:57:15 -08:00
Chris Lu
fc3924ac70 set file name to base file name 
fixing https://github.com/chrislusf/seaweedfs/issues/632
 2018-02-26 08:01:23 -08:00
tengattack
371ad2e397 fix reserve volume on node 2018-01-17 19:53:41 +08:00
Chris Lu
61ff330e99 adjust instructions 2018-01-15 20:33:12 -08:00
Chris Lu
1d09291f95 update upstream 2018-01-15 20:32:57 -08:00
Chris Lu
70f6740309 better fix for single master restart without peers changing 2017-12-06 00:14:14 -08:00
Chris Lu
9026b3e86e always remember the max volume id 2017-11-28 17:08:59 -08:00
Chris Lu
97f757c669 add debug message 
display the error message for debugging
https://github.com/chrislusf/seaweedfs/issues/603
 2017-11-25 01:31:13 -08:00
Chris Lu
ef6224e45d filer default to auto chunking file size to 32MB 2017-09-15 08:24:30 -07:00
Yang Pengcheng
a2112e1be0 fix ldb dir deleting 2017-08-30 22:55:03 +08:00
Chris Lu
3288d729ae fix compilation error 2017-08-30 02:24:19 -07:00
Chris Lu
58344980e4 preallocate disk space during compaction also, add cleanup for failed compaction 2017-08-29 23:59:53 -07:00
Chris Lu
f7c22f0159 fix bug when collecting vacuum errors 2017-08-29 23:11:20 -07:00
chanyoung
52fc476a6f filer: fixup print error message when make new directory 2017-08-15 15:54:12 +09:00
Chris Lu
dbee816205 fix typo 
fix https://github.com/chrislusf/seaweedfs/issues/549
 2017-08-10 10:26:39 -07:00
Chris Lu
3a8746ca54 better error message 2017-08-10 10:26:39 -07:00
kelgon
3bf883327e (fix #543)added body to error when Post encounter 4xx response 2017-07-28 16:11:23 +08:00
Chris Lu
e0dfa7a069 ensure 30GB or less 
avoid https://github.com/chrislusf/seaweedfs/issues/535
 2017-07-16 21:40:55 -07:00
Xinran Xu
4a0939a4e4 Fix panic error when target collection is not exist. 
Fix: interface conversion: interface {} is nil, not *topology.Collection.
 2017-07-14 20:04:33 +08:00
Chris Lu
f9a1769df1 fix nil raft server caused crash 
fix https://github.com/chrislusf/seaweedfs/issues/524
 2017-07-03 22:35:28 -07:00
Chris Lu
1fbb8723dc add cpu and mem pprof 2017-06-22 01:33:58 -07:00
Chris Lu
2c7dad589d stop when dn.parent is nil 
fix https://github.com/chrislusf/seaweedfs/issues/473
 2017-06-16 08:27:50 -07:00
Chris Lu
72e89b615b 301 is reported as 404 for http post 
fix https://github.com/chrislusf/seaweedfs/issues/512
 2017-06-15 21:21:32 -07:00
Chris Lu
5047bdb4a2 skip bytes cache 2017-06-03 11:44:24 -07:00
Chris Lu
f5bed84340 add volume id to error message 2017-06-03 01:58:28 -07:00
Chris Lu
83efe1547d revert webp support due to "go test" 
related to https://github.com/chai2010/webp/issues/9
 2017-05-28 10:25:40 -07:00
Chris Lu
478fe0ecf2 filer add readonly public port 2017-05-27 20:14:22 -07:00
Chris Lu
7ecc0f4b11 rename variables 2017-05-27 18:11:18 -07:00
Chris Lu
021ef0d14b adjust version to 0.76 2017-05-26 23:04:38 -07:00
Chris Lu
e3fc676260 add doc for 0.75 2017-05-26 23:01:40 -07:00
Chris Lu
82c3ccc8dd add btree for volume index 2017-05-26 22:51:25 -07:00
Chris Lu
bd97cbc523 skip busy reporting back to master 
possible fix for https://github.com/chrislusf/seaweedfs/issues/499
 2017-05-23 21:12:03 -07:00
Chris Lu
a7faefd4ce fix compilation error 2017-05-22 23:04:43 -07:00
Chris Lu
b3513d3167 skip only readonly volumes 
fix https://github.com/chrislusf/seaweedfs/issues/405
 2017-05-22 17:05:27 -07:00
Chris Lu
4ce6586710 use local random generator to avoid global lock 2017-05-19 23:33:36 -07:00
rmn
366fe0d394 Scale images to fit or fill 2017-05-05 12:17:30 +03:00
Vitaly Baev
7ab9160ee1 WebP resize support 2017-03-31 09:56:00 +03:00
Chris Lu
a337b844ec properly send heartbeats 
fix https://github.com/chrislusf/seaweedfs/issues/476
 2017-03-27 10:11:54 -07:00
sparklxb
c46e91d229 complement weed mount: add read and delete 2017-03-17 15:02:27 +08:00
Chris Lu
37b79174d1 Merge pull request #469 from sparklxb/master 
complement weed mount
 2017-03-14 08:21:29 -07:00
Stepan Bujnak
14ec3bfc1d Show contents of manifest file when cm=false is passed 2017-03-12 18:40:24 -07:00
Stepan Bujnak
a72c92e55f Added locking on the filer move operation 2017-03-11 02:52:04 -08:00
sparklxb
3a67b6ddc1 complement weed mount 2017-03-10 14:46:00 +08:00
sparklxb
7f85e15990 complement weed mount 2017-03-10 10:28:19 +08:00
sparklxb
813a4f7456 Fixes #266 2017-03-06 13:04:52 +08:00
sparklxb
4049c72e6c fix bug: replication, metadata not replicated 2017-03-04 11:24:12 +08:00
Chris Lu
30f138a538 fix tests 2017-02-12 22:18:48 -08:00
Chris Lu
41f686ab9d remove weeds.conf 2017-02-12 21:58:44 -08:00
Chris Lu
98d66a569b avoid nil 
fix https://github.com/chrislusf/seaweedfs/issues/452
 2017-02-11 10:21:28 -08:00
vancepym
ee6067e98c Fix: the maximum-length character exceed 255 
The length of a CHAR column is fixed to the length that you declare when you create the table. The length can be any value from 0 to 255. When CHAR values are stored, they are right-padded with spaces to the specified length. 

see https://dev.mysql.com/doc/refman/5.7/en/char.html
 2017-01-23 20:33:00 +08:00
Chris Lu
c2ce97785f set the volume size limit 
fixing https://github.com/chrislusf/seaweedfs/issues/447
 2017-01-22 21:36:36 -08:00
Chris Lu
f302c9fda2 always send the leader info on each heartbeat 2017-01-21 13:58:56 -08:00
Chris Lu
3cb0071025 supporting reading file of size zero 2017-01-21 13:45:16 -08:00
Chris Lu
043b7a7c65 add locking to access l.volumes 2017-01-20 10:18:43 -08:00
brstgt
0656838fe5 Don't return actual file size from CheckVolumeDataIntegrity, it will be 0 if last needle is a tombstone, so it's not reliable anyway 2017-01-20 16:37:45 +01:00
brstgt
4fb5bb09b6 Remove obsolete property Volume.dataFileSize 2017-01-20 16:31:11 +01:00
brstgt
e074a54a20 Delete volumes online without restarting volume server 2017-01-20 13:02:37 +01:00
brstgt
492f93416d Mount and unmount volumes online without restarting volume server 2017-01-20 12:49:20 +01:00
brstgt
18b3afc97a Log volumeId if dataFileSize != actual data file size - improve log types 2017-01-20 12:22:25 +01:00
brstgt
0c898df430 Log volumeId if dataFileSize != actual data file size 2017-01-20 11:52:23 +01:00
Chris Lu
c8f54aad8b adjust timing of leader election 2017-01-18 09:54:43 -08:00
Chris Lu
8de0027df5 Release 0.74 2017-01-18 09:39:47 -08:00
Chris Lu
5b0f5f456e volume server get notified of leader change 2017-01-18 09:34:27 -08:00
Chris Lu
59022b6fe0 fix byte counter on loading index file 
fix https://github.com/chrislusf/seaweedfs/issues/441
 2017-01-18 08:40:39 -08:00
Chris Lu
76da296799 upgrade to 0.73 2017-01-15 12:29:53 -08:00
Chris Lu
3065506b38 volume servers always connect to the master leader 2017-01-13 10:20:40 -08:00
Chris Lu
d953ed442c auto detect volume server ip address 2017-01-12 13:42:53 -08:00
Chris Lu
1be354bdce default IP to "localhost" 
so that the master can know the ip address.
 2017-01-10 23:57:59 -08:00
Chris Lu
adc8d344a5 update proto buffer 2017-01-10 23:57:16 -08:00
Chris Lu
e767c3ea4f disable master connection timeout 
temporarily disable master connection timeout due to heartbeat
connection timeout
 2017-01-10 01:30:00 -08:00
Chris Lu
e46c3415f7 gRpc for master~volume heartbeat 2017-01-10 01:01:12 -08:00
Chris Lu
4beaaa0650 simplify the gzip optimization logic 
fix https://github.com/chrislusf/seaweedfs/issues/436
 2017-01-09 19:31:58 -08:00
Chris Lu
9fa648e570 Merge pull request #435 from sparklxb/master 
support uploading files to specific dataCenter
 2017-01-09 10:07:07 -08:00
sparklxb
66e7013dfe suport uploading files to specific dataCenter 2017-01-09 23:34:47 +08:00
wangjie
90a6f43c56 fix the bug than we can't get filename when download file. 2017-01-09 14:55:21 +08:00
Chris Lu
53cf1b4900 change the basic unit to MB 2017-01-08 14:57:32 -08:00
Chris Lu
ed44f12f6d support Fallocate on linux 2017-01-08 11:01:46 -08:00
Chris Lu
7b6837cbc2 move back the section 2017-01-08 10:35:47 -08:00
sparklxb
da9b672d1b support additional header name-value pairs 2017-01-08 22:34:42 +08:00
sparklxb
86a7c56275 support additional header name-value pairs 2017-01-08 09:16:40 +08:00
Chris Lu
13e7069eb9 keep track of total data file size 2017-01-06 10:22:20 -08:00
Chris Lu
e61c9af523 check Accept instead of content type 
fix https://github.com/chrislusf/seaweedfs/issues/429
 2017-01-04 19:58:27 -08:00
Chris Lu
dc42d15352 release 0.72 2017-01-03 21:32:59 -08:00
Chris Lu
4d3353750e fix build issue 2017-01-03 21:28:30 -08:00
Chris Lu
297f64e286 Merge pull request #428 from sparklxb/master 
fix bug: upload big .gz file more than maxMB
 2017-01-03 21:23:08 -08:00
Chris Lu
8fb29e5230 change to 0.71 version 2017-01-03 21:17:54 -08:00
lixianbin
d96d0a87cf fix bug: upload big .gz file more than maxMB 2017-01-04 11:23:40 +08:00
listeng
5b18cf09e7 Return json data when content-type is "application/json" 
Before #344, the filer will return json data, and cschiano make a template that render to html. But sometimes need json data, so I add some code will return json data when content-type is "application/json".
 2016-12-26 16:49:43 +08:00
Mike Tolman
43e655afc8 Updating PostgreSQL file store integration to support directories. 2016-12-12 13:03:32 -07:00
Mike Tolman
e88795be13 Adding PostgreSQL Support 2016-12-07 17:24:40 -07:00
Mike Tolman
b18ec59c56 Revert "Adding PostgreSQL Filer Support" 
This reverts commit 10853e4d2f.
 2016-12-07 17:16:10 -07:00
Mike Tolman
10853e4d2f Adding PostgreSQL Filer Support 2016-12-07 17:13:03 -07:00
Mike Tolman
a1783a14cc Revert "Adding PostgreSQL Filer Support" 
This reverts commit 855058858a.
 2016-12-07 16:54:06 -07:00
Mike Tolman
855058858a Adding PostgreSQL Filer Support 2016-12-07 16:51:36 -07:00
Chris Lu
64509dd6ca fix text. 2016-11-28 20:07:27 -08:00
Chris Lu
924f797c58 Merge pull request #406 from eshujiushiwo/master 
Fix the filer connection of cassandra
 2016-11-28 20:05:45 -08:00
eshujiushiwo
e025fc00a4 let filer use all cassandra server 
let filer use all cassandra server
 2016-11-28 18:23:39 +08:00
eshujiushiwo
5ee7fdc0cc fix hosts strings 
fix hosts strings
 2016-11-28 16:03:39 +08:00
eshujiushiwo
3450eff288 fix info log 
fix info log
 2016-11-28 15:32:16 +08:00
eshujiushiwo
3e29d0f75b Update cassandra_store.go 2016-11-28 15:14:19 +08:00
eshujiushiwo
809aa028ec fix the connection of cassandra 
fix the connection of cassandra ,let filer can connect to multi cassandra nodes which are in the one cluster
 2016-11-28 15:01:04 +08:00
eshujiushiwo
89ccb6be05 Update cassandra_store.go 2016-11-28 12:16:51 +08:00
eshujiushiwo
5385bf9555 update cassandra connections 
update cassandra connections
 2016-11-28 11:27:40 +08:00
Chris Lu
9ba52db585 add writable checking 
fix https://github.com/chrislusf/seaweedfs/issues/405
 2016-11-26 12:14:06 -08:00
Chris Lu
dcaf1796fe add option to enable caching 2016-11-16 07:09:57 -08:00
Chris Lu
f54f530ada adjust verbose logging 2016-11-13 14:07:51 -08:00
霍晓栋
b9f385bd66 refactor concurrent loading code logic 2016-11-13 13:24:52 +08:00
霍晓栋
096ffa9744 concurrent loading volume 2016-11-11 11:53:22 +08:00
Chris Lu
3320e495f4 Merge pull request #394 from bedragon/master 
fix bug : join post is empty
 2016-11-07 19:09:19 -08:00
Yang zhixiang
57005ed7f8 fix bug : join post is empty 2016-11-08 11:00:46 +08:00
Chris Lu
36f9633223 add locks for location.volumes 
fix https://github.com/chrislusf/seaweedfs/issues/392
 2016-11-06 20:55:22 -08:00
Yang zhixiang
4194ae33fd fix raft_server.go isPeersChanged bug 2016-11-03 15:46:59 +08:00
霍晓栋
7d73bbb073 comment UT case 2016-10-07 16:40:51 +08:00
霍晓栋
7d382ba5fe comment UT case 2016-10-07 16:34:22 +08:00
霍晓栋
ce1f7ab662 makediff func with UT case 2016-10-07 16:22:24 +08:00
霍晓栋
ed848425c7 supplemental data between compacting and commit compacting 2016-09-29 13:57:23 +08:00
Chris Lu
dffad65f2f fix compilation 2016-09-26 22:30:44 -07:00
Chris Lu
7e29218327 add a template for makeupDiff 2016-09-26 22:26:41 -07:00
Chris Lu
1bc041b46d add a new way to manually compact corrupted volume 
fix https://github.com/chrislusf/seaweedfs/issues/371
 2016-09-22 20:31:20 -07:00
Chris Lu
01cbd5cb58 lock 
fix https://github.com/chrislusf/seaweedfs/issues/367
 2016-09-08 09:50:31 -07:00
霍晓栋
78474409a5 filer mysqlstore bug fix 2016-09-08 11:35:54 +08:00
霍晓栋
1c2affea8e Merge remote-tracking branch 'upstream/master' 2016-09-08 10:34:24 +08:00
Chris Lu
0559aa9673 use Lock instead of RLock 
fix https://github.com/chrislusf/seaweedfs/issues/364
 2016-09-07 18:21:05 -07:00
Chris Lu
c4b7966dbe minor help message change 2016-09-07 18:21:05 -07:00
霍晓栋
3aa021a812 refactoring mysql store code 2016-09-05 14:10:22 +08:00
霍晓栋
e7b237c8da UT case fix 2016-08-31 11:55:02 +08:00
霍晓栋
b0035747e3 add filer support 2016-08-31 11:32:30 +08:00
霍晓栋
f04d8fcbcc if replicated volume has one copy in readonly mode at one node,it should be removed from writable list 2016-08-09 20:12:39 +08:00
Mike Tolman
01d3f69c52 Adding AutoChunk/MaxMB Support to Filer API 
This is related to the following issue I added to chrislusf/seaweedfs:
https://github.com/chrislusf/seaweedfs/issues/342
 2016-08-05 16:01:30 -06:00
Mike Tolman
a89a3c86d0 Revert "Add AutoChunking to the Filer API, so that you can upload really large files through the filer API." 
This reverts commit 09059bfdcc.
 2016-08-05 15:47:46 -06:00
Mike Tolman
0d331c1e3a Revert "Changing needle_byte_cache so that it doesn't grow so big when larger files are added." 
This reverts commit 87fee21ef5.
 2016-08-05 15:46:45 -06:00
Mike Tolman
0f4c7dd8fd Revert "Ooops. Missed a line." 
This reverts commit 14d4252904.
 2016-08-05 15:46:08 -06:00
Mike Tolman
ce99bb927d Revert "Adding HTTP verb whitelisting options." 
This reverts commit 34837afc7a.
 2016-08-05 15:45:48 -06:00
Mike Tolman
34837afc7a Adding HTTP verb whitelisting options. 2016-08-05 15:23:43 -06:00
Mike Tolman
14d4252904 Ooops. Missed a line. 2016-08-05 15:22:46 -06:00
Mike Tolman
09059bfdcc Add AutoChunking to the Filer API, so that you can upload really large files through the filer API. 2016-08-05 15:19:10 -06:00
Mike Tolman
87fee21ef5 Changing needle_byte_cache so that it doesn't grow so big when larger files are added. 2016-08-05 15:14:24 -06:00
霍晓栋
52e55508da deleted index entry could not point to deleted needle 2016-07-25 15:07:11 +08:00
霍晓栋
b9b3651a98 deleted index entry could not point to deleted needle 2016-07-25 14:56:58 +08:00
霍晓栋
09bd3d015d deleted index entry could not point to deleted needle 2016-07-25 14:54:40 +08:00
霍晓栋
78678f4bcb deleted needle does not need checksum verification 2016-07-25 11:40:35 +08:00
Chris Lu
185a916f5e adjusting command options 2016-07-21 15:00:07 -07:00
Chris Lu
a5be4a6d40 fix package name 2016-07-21 01:23:56 -07:00
Chris Lu
a57162e8bf delete operation does not need this checking 2016-07-21 00:40:13 -07:00
Chris Lu
cdae9fc680 add "weed copy" command to copy files to filer 2016-07-20 23:45:55 -07:00
霍晓栋
574cc41f0b recover filer read handler logic 2016-07-21 12:20:22 +08:00
cschiano
af905a3ff7 Add limit parameter and pagination for files 2016-07-20 10:46:28 +02:00
cschiano
c63cebaee1 Add a visualization for the filer 2016-07-18 10:28:24 +02:00
Chris Lu
d3ae3a8e14 collect proof when stopping 2016-07-05 10:23:14 -07:00
Chris Lu
b38ff39e8b add master profiling option 2016-07-03 15:39:57 -07:00
Chris Lu
d981eb282f refactor volume_read_write.go out of volume.go 2016-07-03 00:10:27 -07:00
Chris Lu
576573711a refactoring 2016-07-02 23:58:23 -07:00
Chris Lu
3d8df0f709 refactor volume_loading.go out of volume.go 2016-07-02 23:56:49 -07:00
Chris Lu
582d5d526c refactoring 2016-07-02 23:50:58 -07:00
Chris Lu
3713119ab9 adjust verification logic. 
1. log file name for easier debugging
2. remove unused version in readIndexEntryAtOffset()
3. if indexSize==0, the dataFile usually still has a super block of
size 8. So skipping this case.
 2016-07-02 23:45:37 -07:00
霍晓栋
16f6984d02 refactor data integrity checking code v2 2016-07-03 14:11:25 +08:00
霍晓栋
dda13def2a refactor data integrity checking code 2016-07-03 12:53:03 +08:00
霍晓栋
2e74fb60c8 add data integrity checking 2016-07-02 21:34:30 +08:00
霍晓栋
339856eb78 fix bug when choosing the healthy master in filer service 2016-06-30 19:54:05 +08:00
霍晓栋
079fdbad13 fix volume_layout bug, add new feature for volume etag write,filer pass through query params' 2016-06-29 16:05:00 +08:00
Chris Lu
b617b13c43 remember oversized volumes 
fix https://github.com/chrislusf/seaweedfs/issues/331
 2016-06-27 15:28:23 -07:00
Chris Lu
d0dbf6d2ea adjust logging 2016-06-26 12:49:10 -07:00
霍晓栋
1f63094542 refactor Volume Assign function 2016-06-26 10:50:18 +08:00
霍晓栋
c601ef03b1 Merge remote-tracking branch 'upstream/master' 2016-06-26 10:27:50 +08:00
Chris Lu
0647c75a06 update dependency versions 2016-06-24 14:20:32 -07:00
Chris Lu
101e784577 add locking on possible concurrent map access 
fix https://github.com/chrislusf/seaweedfs/issues/328
 2016-06-23 09:10:25 -07:00
霍晓栋
79b0a58132 add options params for the Assign func 2016-06-23 11:19:09 +08:00
Chris Lu
356b8048c5 fix compilation problem due to API changes 2016-06-18 18:57:33 -07:00
Chris Lu
cc01c352c3 refactoring 2016-06-13 15:07:17 -07:00
Chris Lu
a61453d3ec fix nil crash 
fix https://github.com/chrislusf/seaweedfs/issues/320
 2016-06-13 14:38:35 -07:00
DaLin
fa2a7b5f2f fix bug: filer DELETE 2016-06-11 01:16:19 +08:00
霍晓栋
3a25af223f refactor work for filer write handler 2016-06-08 15:46:14 +08:00
霍晓栋
26bb9094d4 made a little change for Amazon S3 API's compatibility 2016-06-08 12:30:20 +08:00
Chris Lu
0e3140c54c a little refactoring 2016-06-06 23:50:27 -07:00
霍晓栋
d514b6f3f9 read write handler update 2016-06-07 11:45:00 +08:00
霍晓栋
1ef81ac518 filer could detect master nodes healthy status and choose a working one when encountering SPOF 2016-06-07 11:38:40 +08:00
Chris Lu
96d2f11f81 adjust log 2016-06-05 09:05:03 -07:00
Chris Lu
b456173eb8 delete file storage when POST to update 
PUT is used to update. But sometimes POST is also used to post the same
file again. In this case, the actual file storage was not marked as
deleted.

This fix will try its best to delete the orphan files.
 2016-06-03 00:24:55 -07:00
Chris Lu
535b1610c8 add fid and url to filer post result 
fix https://github.com/chrislusf/seaweedfs/issues/274
 2016-06-02 20:44:50 -07:00
Chris Lu
4f47b8cff9 refactoring 2016-06-02 20:05:34 -07:00
Chris Lu
5ce6bbf076 directory structure change to work with glide 
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
 2016-06-02 18:09:14 -07:00
Chris Lu
5071f528f6 testing compilation with remove package 2013-02-10 03:49:51 -08:00
Chris Lu
55f2627fcf testing new directory structure 2013-02-10 03:31:40 -08:00
Chris Lu
ab6fb13ad7 avoid the "src" folder 2013-02-10 03:25:35 -08:00