mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-12 08:42:01 +08:00
Code Issues Packages Projects Releases Wiki Activity
3,442 Commits 32 Branches 296 Tags 150 MiB
4f02f7121d
Commit Graph

278 Commits

Author SHA1 Message Date
Chris Lu
e6ee421d61 fix compilation 2019-12-14 12:04:25 -08:00
Chris Lu
bbb6ebc3c0 filer: DeleteFolderChildren for deleting large folders 2019-12-13 00:23:05 -08:00
Chris Lu
d0b423bbc0 filer: increase directory listing pagination size 2019-12-12 09:11:10 -08:00
Chris Lu
f51a6d8ba5 increase filer pagination limit 2019-12-11 22:21:37 -08:00
Chris Lu
40ff4eea14 fix nil problem 2019-12-10 23:13:14 -08:00
Chris Lu
0eda75fa2c add logging 2019-12-07 07:56:05 -08:00
Chris Lu
46294b42e6 filer: support memsql 
fix https://github.com/chrislusf/seaweedfs/issues/1141
 2019-11-27 12:34:03 -08:00
Chris Lu
0fca1ae776 adjust build 2019-11-22 23:19:49 -08:00
Chris Lu
bc6cd93431 skip build for 32 bit computers 2019-11-22 22:52:21 -08:00
Chris Lu
253dd20bc6 go fmt 2019-10-23 22:25:59 -07:00
Chris Lu
c0776aa40f trying to compile 2019-10-23 01:27:08 -07:00
Chris Lu
23240efa63 filer: add tikv support 2019-10-23 00:31:31 -07:00
Chris Lu
ae53f63680 filer: recursive deletion optionally ignoring any errors 
fix https://github.com/chrislusf/seaweedfs/issues/1062
 2019-09-11 20:26:20 -07:00
Feng Ye
fc0498798b Add etcd filestore 2019-08-01 10:16:45 +08:00
Chris Lu
8afd8d35b3 master: followers can also lookup and redirect 
improve scalability
 2019-07-28 03:58:13 -07:00
divinerapier
9e3b77c3ad avoid double warpping 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-07-24 16:41:45 +08:00
Chris Lu
862c2cb8e6 go fmt 2019-07-21 21:51:38 -07:00
Chris Lu
7872fc38ea filer leveldb2: correct directory listing 
fix https://github.com/chrislusf/seaweedfs/issues/1001
 2019-07-07 20:37:48 -07:00
Chris Lu
fe2882ce7d fix test 2019-07-02 21:42:06 -07:00
Chris Lu
36250154cc remove println 2019-07-02 21:33:23 -07:00
Chris Lu
4d579f7f57 adjust 2019-07-02 21:28:51 -07:00
Chris Lu
dcc331cf5a parallelize leveldb 2019-07-02 21:25:53 -07:00
Chris Lu
a14bd31f5b go fmt 2019-06-22 20:05:25 -07:00
Chris Lu
3fa1f150d9 refactoring 2019-06-22 20:04:56 -07:00
Chris Lu
1babec00e7 check deleted chunks faster 2019-06-22 13:22:22 -07:00
Chris Lu
c369e5a13b add metrics for filer store 2019-06-22 12:23:25 -07:00
Chris Lu
a111f26fe6 avoid nil 
fix https://github.com/chrislusf/seaweedfs/issues/988
 2019-06-21 20:56:27 -07:00
Chris Lu
308102f023 fix cassandra creation problem 
fix https://github.com/chrislusf/seaweedfs/issues/986
 2019-06-21 17:10:38 -07:00
Chris Lu
059ef879a8 fix issue 986 
fix issue 986
 2019-06-21 13:06:04 -07:00
Chris Lu
11be0b5e91 more logging 2019-06-20 23:45:30 -07:00
Chris Lu
ede876cfdb periodic scripts exeuction from leader master 2019-06-05 01:30:24 -07:00
Chris Lu
3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu
54b835e1ae filer: redis cluster add option for password 
fix https://github.com/chrislusf/seaweedfs/issues/971
 2019-05-20 09:00:30 -07:00
Chris Lu
8123a488cd filer: optimize leveldb parameters 2019-05-17 18:14:25 -07:00
Chris Lu
0fe286a6ca filer: optimization for leveldb, add leveldb2 
large filer db may see high CPU and disk usage due to background compaction
 2019-05-17 17:33:49 -07:00
Chris Lu
bf9d490538 fix for tests 2019-05-17 02:28:20 -07:00
Chris Lu
82b0759493 filer: migrating filer store from persisting shorter structured file id instead of a string 2019-05-17 02:03:23 -07:00
Chris Lu
f0f981e7c8 webdav: fix directory renaming 2019-05-03 14:12:51 -07:00
Chris Lu
bfbecd7253 webdav: fix directory creation 2019-05-03 13:13:11 -07:00
Chris Lu
7064b2ea48 webdav: can write now 2019-05-03 00:55:52 -07:00
Chris Lu
b30c14b631 webdav: can read now 2019-05-03 00:24:35 -07:00
Chris Lu
108d0fb08d adjust log level 2019-04-26 09:32:37 -07:00
Chris Lu
07091c44cd lock btree writes 
fix https://github.com/chrislusf/seaweedfs/issues/930
 2019-04-16 19:48:07 -07:00
Chris Lu
4393b99332 add notes 2019-04-16 13:57:45 -07:00
Chris Lu
338e6d60a5 refactor: prepare for snapshotting 2019-04-16 09:55:37 -07:00
Chris Lu
b3b42bc947 replicate need to include new entry path 2019-04-16 00:44:31 -07:00
Chris Lu
35aba35c07 Revert "weed volume: disable block cache on leveldb" 
This reverts commit a55e708d36.
 2019-04-09 00:22:40 -07:00
Chris Lu
a55e708d36 weed volume: disable block cache on leveldb 
saving 8MB for each volume, but at the cost of slower (offset,size) lookup
 2019-04-09 00:18:32 -07:00
Chris Lu
beaa2bd71a weed filer: properly delete cached directory entry 
fix https://github.com/chrislusf/seaweedfs/issues/910
 2019-04-01 17:03:04 -07:00
Chris Lu
2a52e70a41 weed filer: increase default mysql/postgres dir or name length to 65535 2019-04-01 12:37:54 -07:00
Chris Lu
b4c276263f weed filer: mysql/postgres use ReadCommitted transaction isolation level for atomic rename 2019-04-01 11:03:04 -07:00
Chris Lu
78ac2bef3c go fmt 2019-03-30 23:09:16 -07:00
Chris Lu
97406333a5 support atomic renaming for mysql/postgres filer store 2019-03-30 23:08:29 -07:00
Chris Lu
b92122b885 fix test 2019-03-16 09:50:57 -07:00
Chris Lu
36b632ebcb fix tests 2019-03-16 09:50:21 -07:00
Chris Lu
55bab1b456 add context.Context 2019-03-15 17:20:24 -07:00
Chris Lu
cece860bfd add context to all filer APIs 2019-03-15 15:55:34 -07:00
Chris Lu
55c85f3a66 adding context 2019-03-15 15:26:09 -07:00
Chris Lu
d312c55bbe file path supports windows, avoiding back slashes 
fix https://github.com/chrislusf/seaweedfs/issues/868
 2019-03-04 13:00:08 -08:00
Chris Lu
2442d56671 adjust package name 2019-02-18 18:03:16 -08:00
Chris Lu
77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
bingoohuang
ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
Chris Lu
2a75a36b27 set cassandra ttl with entry.TTLSec 2019-01-08 20:26:19 -08:00
byungjo.yoon
58fd2e5a19 Set redis key ttl with entry.TTLSec 2019-01-09 11:48:19 +09:00
Chris Lu
be9a7592a1 incrementally calculate visible intervals 2019-01-05 22:23:44 -08:00
Chris Lu
832ac8129a reduce pointer usage 2018-12-31 15:10:14 -08:00
Chris Lu
0388d421d2 caching visible intervals for read 
speeds up 4x in single thread mode
speeds up 30% in 32 threads mode
 2018-12-30 00:51:44 -08:00
Chris Lu
253f190f48 weed mount add symlink support 2018-12-25 22:45:44 -08:00
Chris Lu
3ac54792e1 paginate when filer deleting and FUSE mount renaming 2018-12-16 23:20:08 -08:00
Chris Lu
1bfb96f34d optimization for reading whole chunk with gzip encoding 2018-12-07 01:57:55 -08:00
Chris Lu
c28e8a2397 refactoring 2018-12-06 00:44:41 -08:00
Chris Lu
63619f8b8e fail early if two filer stores are defined 2018-12-06 00:32:36 -08:00
Chris Lu
52b24a9902 default "weed server -filer" to same directory as -mdir 2018-12-05 23:24:25 -08:00
Chris Lu
b0838db5a8 support hdfs user name and group names 2018-12-04 00:13:40 -08:00
Chris Lu
c6b646e92c skip creation for root 2018-12-02 23:16:17 -08:00
Chris Lu
f0677c5af1 default root to 0755 2018-12-02 19:59:47 -08:00
Chris Lu
db22666a30 add default root directory 2018-12-02 19:42:50 -08:00
Chris Lu
7a6f49cd42 refactor a bit more 
fix https://github.com/chrislusf/seaweedfs/issues/777
 2018-11-29 00:07:54 -08:00
Chris Lu
ceca078acb avoid overwriting file or directory 
fix https://github.com/chrislusf/seaweedfs/issues/777
 2018-11-29 00:00:56 -08:00
Chris Lu
86dd933596 go fmt 2018-11-23 00:26:15 -08:00
Chris Lu
99256100a0 fix tests 2018-11-21 16:35:27 -08:00
Chris Lu
e5ad2223a1 add missing intervals 
fix https://github.com/chrislusf/seaweedfs/issues/774
 2018-11-21 16:25:13 -08:00
Chris Lu
b282e34dc2 async file chunk deletion 2018-11-20 20:56:28 -08:00
Chris Lu
b318167cd3 batch delete file ids 2018-11-20 11:35:08 -08:00
Chris Lu
ee3fe07acf fix sort bug 2018-11-18 21:59:53 -08:00
Chris Lu
469863f7f0 merge interval now is quite faster with reusing slices 2018-11-18 21:24:58 -08:00
Chris Lu
d2924c7748 merge intervals is a bit faster 2018-11-18 20:31:39 -08:00
Chris Lu
cb5a10c6a3 adjust the tests 2018-11-18 17:05:36 -08:00
Chris Lu
f1db22d48b add benchmark test for merging intervals 2018-11-18 16:49:08 -08:00
Chris Lu
4fcfc9410f cleanup 2018-11-18 11:51:38 -08:00
Chris Lu
9655dc9ca9 simpler logic 2018-11-18 10:07:30 -08:00
Chris Lu
a64613172d bootstrap filer replication with weed filer.export -targetStore=notification 2018-11-03 12:43:45 -07:00
Chris Lu
4c97ff3717 support AWS SQS as file change notification message queue 2018-10-31 01:11:19 -07:00
Chris Lu
ff66269b62 use grpc to replace http APIs for batch volume id lookup and batch delete 
1. remove batch volume id lookup http API /vol/lookup
2. remove batch delete http API /delete
 2018-10-14 00:12:28 -07:00
Chris Lu
56a5d5af8d ignore duplicated key error if the store has duplicated primary key prevention 
fix https://github.com/chrislusf/seaweedfs/issues/741
 2018-09-30 00:49:52 -07:00
Chris Lu
7bb62b9f21 delay file creation without chunks from dir.Create to file.Flush 2018-09-22 00:11:46 -07:00
Chris Lu
db69ce89f0 go fmt 2018-09-21 01:56:43 -07:00
Chris Lu
788acdf527 add WIP filer.replicate 2018-09-17 00:27:56 -07:00
Chris Lu
70e67b7c6d add notification log 2018-09-16 11:20:36 -07:00
Chris Lu
d923ba2206 renaming msgqueue to notification 2018-09-16 01:18:30 -07:00
Chris Lu
296acc4a0a go fmt 2018-09-09 16:26:11 -07:00
Chris Lu
164091c269 add s3 multipart upload 2018-09-09 16:25:43 -07:00
Chris Lu
b23883b3b6 fix a bug on error reporting 2018-08-22 18:19:26 -07:00
Chris Lu
f22d6b87e1 go fmt 2018-08-19 15:42:50 -07:00
Chris Lu
748870b03f add "weed scaffold" command 2018-08-19 15:36:30 -07:00
Chris Lu
95b997fb35 go fmt 2018-08-19 15:18:37 -07:00
Chris Lu
f827ada811 merge notification config with filer.toml 2018-08-19 15:17:55 -07:00
Chris Lu
ee920d4f83 kafka set the right topic 2018-08-19 01:27:30 -07:00
Chris Lu
708acee502 add support for redis cluster 
fix https://github.com/chrislusf/seaweedfs/issues/705
 2018-08-15 00:01:38 -07:00
Chris Lu
ae9b81c18f fix tests 2018-08-13 01:33:21 -07:00
Chris Lu
f036ef8a3c add filer notification 2018-08-13 01:20:49 -07:00
Chris Lu
c81f1cda47 correctly recursively delete folders 2018-08-01 01:26:41 -07:00
Chris Lu
d3205a0070 go fmt 2018-07-28 21:02:56 -07:00
Chris Lu
888eb2abb5 filer read write all via locations from MasterClient 2018-07-28 14:51:36 -07:00
Chris Lu
1d779389cb MasterClient replicates all vid locations 2018-07-28 14:22:46 -07:00
Chris Lu
01bcc89803 refactor into MasterClient 2018-07-28 02:10:32 -07:00
Chris Lu
a12c7b86b0 broadcast messages of new and deleted volumes 2018-07-27 23:09:55 -07:00
Chris Lu
6f30a78a6c update also delete old chunks if different from latest chunks 2018-07-21 17:40:00 -07:00
Chris Lu
7e2031b18f go fmt 2018-07-21 17:39:10 -07:00
Chris Lu
80d80daf64 set filer2.ErrNotFound for not found entry 2018-07-20 02:14:18 -07:00
Chris Lu
d429b6296b fix test 2018-07-19 01:27:45 -07:00
Chris Lu
c7228fefa7 add bucket creation and deletion 
1. option for "weed s3 -filer.dir.buckets" to choose a folder for buckets
2. create a bucket
3. delete a bucket, recursively delete all metadata on filer
 2018-07-19 01:21:44 -07:00
Chris Lu
9e974f6a1b
Merge pull request #686 from qszxnp/filer2_dir_cache_bugfix 
bugfix:delete Directory cache when filer2 delete Directory
 2018-07-12 01:17:32 -07:00
聂鹏
1ed41f8b21 bugfix:check Directory attr before create it 2018-07-12 00:53:31 -04:00
聂鹏
c523bc4f07 bugfix:delete Directory cache when filer2 delete Directory 2018-07-11 21:27:25 -04:00
Chris Lu
77fc8c5914 keep alive for gRpc calls 2018-07-03 19:07:55 -07:00
Chris Lu
14c6f12b0c adjust interface parameter name and doc 2018-06-17 13:24:57 -07:00
Chris Lu
db0c25b3d8 delete old entry only after successful insert 2018-06-17 13:02:22 -07:00
Chris Lu
eb2acd11c2 decouple from viper for filer store 2018-06-17 13:01:57 -07:00
Chris Lu
98110c1697 add collection, replication, ttl to each file 2018-06-10 16:57:32 -07:00
Chris Lu
5c25d29272 support renaming files 2018-06-06 22:11:01 -07:00
Chris Lu
43e3f5724c use fixed list of masters in both filer and volume servers 2018-06-01 00:39:39 -07:00
Chris Lu
0301504184 add mime, use simple insert and update filer store API 
1. add mime type to file in filer
2. purge old chunks if overwrite during insert
 2018-05-30 20:24:57 -07:00
Chris Lu
74332e1a61 minor 2018-05-28 14:32:16 -07:00
Chris Lu
07e0d13d2d filer support reading multiple chunks, with range support 2018-05-28 05:39:12 -07:00
Chris Lu
8db9319a06 refactoring: go fmt, reorg 2018-05-27 23:59:49 -07:00
Chris Lu
8b0718ac92 go vet 2018-05-27 11:56:49 -07:00
Chris Lu
458ada173e go fmt 2018-05-27 11:52:26 -07:00
Chris Lu
03d76479a5 add reds 2018-05-27 11:14:29 -07:00
Chris Lu
a6a6b87f1b adjust readme 2018-05-27 11:14:12 -07:00
Chris Lu
ab86d263c9 add Cassandra 2018-05-27 00:01:15 -07:00
Chris Lu
603de2d5db skip deletion if entry not found 2018-05-27 00:00:56 -07:00
Chris Lu
f124ebab1d fix ordering 2018-05-27 00:00:11 -07:00
Chris Lu
869161a261 support both mysql and postgres 2018-05-26 22:02:49 -07:00
Chris Lu
87b3b84471 simplifying dirhash to 64bit integer 2018-05-26 21:24:03 -07:00
Chris Lu
955eae3500 this works great and support long file names 2018-05-26 14:08:55 -07:00
Chris Lu
2da84ed331 working fine now 
but index length can be improved
 2018-05-26 13:35:56 -07:00
Chris Lu
68bcaff14d mysql can compile, not yet tested! 2018-05-26 05:32:15 -07:00
Chris Lu
c1e353491a refactoring variable names 2018-05-26 04:50:55 -07:00
Chris Lu
828e4a5ace fix test error 2018-05-26 04:13:50 -07:00