mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-11-28 21:39:02 +08:00
Code Issues Packages Projects Releases Wiki Activity
4,541 Commits 32 Branches 296 Tags 148 MiB
6856b0d57e
Commit Graph

801 Commits

Author SHA1 Message Date
Chris Lu
86d4b18eb5 filer: optional recursive deletion 
fix https://github.com/chrislusf/seaweedfs/issues/1176
 2019-12-31 11:52:54 -08:00
Chris Lu
509f314350 keep .vif file for ec to normal volume conversion 2019-12-28 21:52:55 -08:00
Chris Lu
0cc8347a3a go fmt 2019-12-28 21:13:10 -08:00
Chris Lu
8fbc0a9163 fix edge cases 2019-12-28 12:59:31 -08:00
Chris Lu
37b64a50b4 ec: generate and copy .vif file 2019-12-28 12:44:59 -08:00
Chris Lu
2000284435 rename volume tier info to volume info 2019-12-28 11:21:49 -08:00
Chris Lu
b3c83f9404 fix spelling 2019-12-28 11:16:10 -08:00
Chris Lu
48d28d3eb2 tier: support remote file attributes and remember the file extension 2019-12-25 21:37:24 -08:00
Chris Lu
d960b3474a tier storage: support downloading the remote dat files 2019-12-25 09:53:13 -08:00
Chris Lu
7ba6be2312 volume ui add readonly status 2019-12-24 14:55:26 -08:00
stlpmo
f87ccbddb4 remove the http 304 response body 2019-12-24 17:56:50 +08:00
Chris Lu
8a2d529759 go fmt 2019-12-23 12:48:53 -08:00
Chris Lu
09ca936c78 shell: add ec.decode command 2019-12-23 12:48:20 -08:00
Chris Lu
36ddca9d1f filer: return not found 404 status if not found 
fix https://github.com/chrislusf/seaweedfs/issues/1160
 2019-12-18 21:04:40 -08:00
Chris Lu
58f88e530c volume: use sorted index map for readonly volumes 2019-12-18 01:21:21 -08:00
Chris Lu
014487b1a6 passing entry.Extended 2019-12-17 21:39:48 -08:00
Chris Lu
e707211dad passing entry.exended 2019-12-17 21:22:16 -08:00
Chris Lu
ad9d5ff5d0 adjust UI 2019-12-15 20:57:17 -08:00
Chris Lu
392678f8f3 upload skipping mimetype if not needed 2019-12-15 20:57:08 -08:00
Chris Lu
bbb6ebc3c0 filer: DeleteFolderChildren for deleting large folders 2019-12-13 00:23:05 -08:00
Chris Lu
0fa1269bc7 filer: streaming file listing 2019-12-13 00:22:37 -08:00
Chris Lu
d0b423bbc0 filer: increase directory listing pagination size 2019-12-12 09:11:10 -08:00
Chris Lu
1f400489c6 filer: UI ensure a correct foler link 2019-12-11 23:09:01 -08:00
Chris Lu
f51a6d8ba5 increase filer pagination limit 2019-12-11 22:21:37 -08:00
Chris Lu
1ddfbff2d3 deletion add option skipChunkDeletion 2019-12-11 14:58:22 -08:00
Chris Lu
f06b65a6fd filer: EXPERT add shouldDeleteChunks parameter for fast deletion 2019-12-11 07:50:53 -08:00
Yoni Nakache
9e45702334
Filer: Enable delete entries without flag volume 
same as happens on the s3 API on delete bucket,
to reduce the IO requests on the system, you can just clear recursively folders without flagging them in volume server (they will be removed on delete bucket/drop collection)
this will remove the user ability to retrieve data under these folders
 2019-12-11 11:30:19 +02:00
Chris Lu
10bd3c6b4b refactoring 2019-12-08 19:44:16 -08:00
Chris Lu
e2a8cdfc48 webdav: fix writing file 
fix https://github.com/chrislusf/seaweedfs/issues/1114
 2019-12-07 07:56:51 -08:00
Chris Lu
0eda75fa2c add logging 2019-12-07 07:56:05 -08:00
Chris Lu
81f424a630 go fmt 2019-12-06 06:57:32 -08:00
Chris Lu
916423e96a master: submit support "dataCenter" parameter 2019-12-05 15:51:24 -08:00
Chris Lu
e426bd541e grow volumes on volume servers with slots freed by cloud storage 2019-12-03 21:36:42 -08:00
Chris Lu
1becbce657 display remote volumes on volume server ui page 2019-12-02 23:23:54 -08:00
Chris Lu
6383b45bd0 add lock variable 2019-12-02 20:49:50 -08:00
Chris Lu
ec8de250e2 tiered storage: can copy to s3, read from s3 
master not aware tiered volume yet, file assigning is not working yet
 2019-12-02 15:08:31 -08:00
Chris Lu
0da7b894cc pass backend config from master to volume servers 2019-11-29 01:05:09 -08:00
Chris Lu
ea9a7e61d0 only send metrics address during on the first response 2019-11-28 20:48:08 -08:00
Chris Lu
f60154f330 master load backend storage config from master.toml 2019-11-28 18:33:18 -08:00
Chris Lu
586798ecc0 go fmt 2019-11-27 12:34:57 -08:00
Chris Lu
0f9ba84274 s3 2019-11-27 03:09:45 -08:00
Lei Liu
2700ac6d1f filer: Add attr.mode in query string 
Signed-off-by: Lei Liu <liul.stone@gmail.com>
 2019-11-27 13:57:29 +08:00
Chris Lu
fb3c9ed80d shell: fs.meta.save missed directory data with more than 1024 entries 
fix https://github.com/chrislusf/seaweedfs/issues/1130
 2019-11-22 01:39:50 -08:00
Chris Lu
de0c983808 refactoring scaffolding for master.toml 2019-11-14 22:20:38 -08:00
Lei Liu
46755ea1e1 fix master maintenance logic 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-11-12 14:46:10 +08:00
Chris Lu
ee90236a97 Merge branch 'master' into refactoring_dat_backend 2019-11-11 21:30:14 -08:00
Chris Lu
d7852ebb3b
Merge pull request #1102 from stlpmo-jn/add_ETCD_sequencer 
Add etcd sequencer
 2019-11-11 12:43:24 -08:00
stlpmo
62d393d6c9 ut pass 2019-11-11 10:52:21 +08:00
stlpmo
802a0eb3fe move from cmd-line to scaffold 2019-11-11 09:15:17 +08:00
zhangsong
61fa485700 add volume number param in assign operation 2019-11-10 20:11:03 +08:00
Chris Lu
c5c1d83d91 Merge branch 'master' into refactoring_dat_backend 2019-11-08 22:54:41 -08:00
Chris Lu
6cc9e0d788 volume: fix ec shard reading 
fix https://github.com/chrislusf/seaweedfs/issues/1106
 2019-11-08 22:41:02 -08:00
stlpmo
1c8bed3810 delete the var etcdClient and comments 
refactor the code
add sequencer cmd-line
delete nerver used codes
 2019-11-05 15:45:25 +08:00
Lei Liu
1294999d8b return error when garbageThreshold is not a valid float number 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-31 20:43:20 +08:00
Lei Liu
1bcef02a6c fix dir/lookup and col/delete api 
1, Fix Layouts first letter capitalized
2, Return http 204 when delete a collection

Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-31 16:55:45 +08:00
Lei Liu
f2f90436ef fix leader master /dir/lookup api 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-30 16:38:40 +08:00
Chris Lu
5b950c735e Merge branch 'master' into refactoring_dat_backend 2019-10-29 23:18:41 -07:00
Lei Liu
c2884cace2 misc updated 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-29 21:28:28 +08:00
Chris Lu
19b6a16003 changed from os.file to backend.DataStorageBackend 2019-10-29 00:35:16 -07:00
Chris Lu
71eb8efd20 fix error logs 2019-10-25 09:44:58 -07:00
Chris Lu
55a2221eba master: add logging for volume server disconnection 
add logging for https://github.com/chrislusf/seaweedfs/issues/1093
 2019-10-25 09:35:19 -07:00
Chris Lu
46ed2ca902 refactoring 2019-10-24 23:41:32 -07:00
divinerapier
f1ee6b7a54
fix abused 404 status code 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-10-24 16:51:26 +08:00
Chris Lu
c358f96aaa volume: redirection follow public url 
fix https://github.com/chrislusf/seaweedfs/issues/1089
 2019-10-23 22:11:42 -07:00
Chris Lu
23240efa63 filer: add tikv support 2019-10-23 00:31:31 -07:00
Chris Lu
c71fddee2b go fmt 2019-10-22 00:50:50 -07:00
Chris Lu
fc412e428b refactor ScanVolumeFileFrom() 2019-10-22 00:50:30 -07:00
Chris Lu
c9a183eb69 refactor memory map related code 2019-10-22 00:49:42 -07:00
Chris Lu
faec9076a4 adjust parameter names 2019-10-21 22:57:01 -07:00
Chris Lu
d16c450682 fix unreachable code 2019-10-21 22:36:41 -07:00
joeslay
d53aee179b
Merge pull request #10 from chrislusf/master 
merge seaweed master
 2019-10-14 16:03:40 +01:00
Chris Lu
939e4b57a8 go fmt 2019-10-09 00:03:18 -07:00
Chris Lu
f8d4b7d1c0 support basic json filtering and selection 2019-10-06 22:35:08 -07:00
Chris Lu
cf47f657af scaffold for volume server query feature 2019-10-06 22:35:08 -07:00
joeslay
9d03aa6a38
Merge pull request #8 from chrislusf/master 
merge seaweed master
 2019-10-01 17:55:31 +01:00
Lei Liu
3cc084269c master api: return http 404 when volumeId not exist 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-09-29 14:23:39 +08:00
j.laycock
6fc6322c90 Change joeslay paths to chrislusf paths 2019-09-12 14:18:21 +01:00
joeslay
d8c34b032f
Merge pull request #6 from chrislusf/master 
merge seaweed master
 2019-09-12 11:44:09 +01:00
Chris Lu
ae53f63680 filer: recursive deletion optionally ignoring any errors 
fix https://github.com/chrislusf/seaweedfs/issues/1062
 2019-09-11 20:26:20 -07:00
Tom Maxwell
4a878c0006 Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve 2019-09-04 15:27:14 +01:00
joeslay
6ee65356e3
Merge pull request #3 from chrislusf/master 
merge seaweed master
 2019-09-04 14:58:04 +01:00
j.laycock
cbd0a98fa1 Add InMemory to Volume Grow structure 2019-09-03 18:19:02 +01:00
Chris Lu
ee8bcb0c12 remove println 2019-09-03 09:22:56 -07:00
j.laycock
1f01eb78e8 Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct 2019-09-03 17:00:59 +01:00
Tom Maxwell
d637d86d22 Changes to try and pass the URL parameters through - in memory flag not working still 2019-09-03 15:41:28 +01:00
joeslay
e91139348e
Merge pull request #2 from chrislusf/master 
merge seaweed master
 2019-09-02 11:39:40 +01:00
j.laycock
595a1beff0 Swap imports to use joeslay 2019-09-02 11:28:40 +01:00
Chris Lu
60c9215a00 HCFS: avoid verbose file not found exception stack trace 2019-09-01 21:40:26 -07:00
Chris Lu
0dc43ecdca skip writing content if not modified 
related to https://github.com/chrislusf/seaweedfs/issues/1036
 2019-08-09 14:38:54 -07:00
Feng Ye
fc0498798b Add etcd filestore 2019-08-01 10:16:45 +08:00
Chris Lu
ca2b81a944 fix test error 2019-07-31 02:09:13 -07:00
Chris Lu
19360ab10a master: redirect clients to the new leader 2019-07-31 01:54:42 -07:00
Chris Lu
8afd8d35b3 master: followers can also lookup and redirect 
improve scalability
 2019-07-28 03:58:13 -07:00
Chris Lu
898d943b25 refactoring 2019-07-17 23:43:48 -07:00
Chris Lu
0264a7f50a set the max file key when ever possible 2019-07-17 23:23:01 -07:00
Chris Lu
aff911c00d skip all azuresb 2019-07-17 01:24:20 -07:00
Chris Lu
0ea98dc6a2 disable azure due to api changes 2019-07-17 00:39:36 -07:00
Chris Lu
9e47c4749a shell: add fs.mv 
fix https://github.com/chrislusf/seaweedfs/issues/954
 2019-07-11 09:29:04 -07:00
Chris Lu
1d1b355f9e filer: set file name when uploading to a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1005
 2019-07-11 04:20:03 -07:00
Chris Lu
476a7c269c detect a filer path is a file or a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1004
 2019-07-10 12:11:18 -07:00
Chris Lu
15ecf4e3bf filer: default to leveldb2 2019-06-30 00:44:57 -07:00
Chris Lu
6883f9e322 mark volume readonly before ec encoding 2019-06-26 23:02:22 -07:00
Chris Lu
359563e95a refactoring 2019-06-25 09:49:27 -07:00
Chris Lu
c777102da9 go fmt 2019-06-23 15:30:16 -07:00
Chris Lu
d5560f2705 simplify metrics settings 2019-06-23 15:29:49 -07:00
Chris Lu
f16375621f big refactoring 2019-06-23 03:08:27 -07:00
Chris Lu
6c01fb6d2d go fmt 2019-06-23 01:57:51 -07:00
Chris Lu
8da5d5b094 filer.copy: use filer settings, avoid unnecessary command line options 
fix https://github.com/chrislusf/seaweedfs/issues/968
 2019-06-23 01:57:35 -07:00
Chris Lu
d1cd8f8c5b add metrics, refactoring 2019-06-22 22:53:52 -07:00
Chris Lu
3fa1f150d9 refactoring 2019-06-22 20:04:56 -07:00
Chris Lu
f88a8bda7b ec deletion works 2019-06-21 01:14:10 -07:00
Chris Lu
613a2e8060 delete garbage only when successful 2019-06-20 23:46:00 -07:00
Chris Lu
c74dc2b306 go fmt 2019-06-20 00:55:52 -07:00
Chris Lu
e63317fb08 ec deletion code complete, not tested yet 2019-06-20 00:55:30 -07:00
Chris Lu
4cea8aefd0 add grpc VolumeEcBlobDelete 2019-06-20 00:17:11 -07:00
Chris Lu
856da7aae2 ec volume support deletes 2019-06-19 22:57:14 -07:00
Chris Lu
d8ed73926d volume servers get metrics address and interval from the master 2019-06-17 14:51:47 -07:00
Chris Lu
ff6a6dd11e refactoring 2019-06-15 21:46:55 -07:00
Chris Lu
8b43679ae3 refactoring 2019-06-15 12:21:44 -07:00
Chris Lu
5336008dcd refactoring 2019-06-14 13:06:01 -07:00
Chris Lu
d67189921f go fmt 2019-06-14 00:55:12 -07:00
Chris Lu
5f6c9825f8 volume server: adds basic metrics 2019-06-14 00:54:56 -07:00
Chris Lu
a11525fe4e filer: adds basic metrics pushing to Prometheus gateway 2019-06-13 02:01:54 -07:00
Chris Lu
57092d69ac go fmt 2019-06-10 21:33:32 -07:00
Chris Lu
e66bddd84c customizable sleep intervals 2019-06-06 00:39:08 -07:00
Chris Lu
50aa769554 jwt for read access control 2019-06-06 00:29:02 -07:00
Chris Lu
d344e0a035 fix ec related bugs 2019-06-05 23:20:26 -07:00
Chris Lu
450f4733ce report empty volume or ec shards 2019-06-05 13:32:33 -07:00
Chris Lu
7f6fb9ffd0 every 17 minutes instead of seconds 
TODO: make this configurable
 2019-06-05 13:32:08 -07:00
Chris Lu
ede876cfdb periodic scripts exeuction from leader master 2019-06-05 01:30:24 -07:00
Chris Lu
ca8a2bb534 go fmt 2019-06-04 22:04:10 -07:00
Chris Lu
2215e81be7 ui add ec shard statuses 2019-06-04 21:52:37 -07:00
Chris Lu
b05456fe07 able to purge extra ec shard copies 2019-06-03 20:25:02 -07:00
Chris Lu
11cffb3168 fix ec.rebuild bugs 2019-06-03 11:50:54 -07:00
Chris Lu
d85b41b904 fix ec.encode not finding the local ec shards 2019-06-03 10:38:21 -07:00
Chris Lu
7e80b2b882 fix multiple bugs 2019-06-03 02:26:31 -07:00
Chris Lu
55be09996d fix volume balance bug 2019-06-03 00:13:31 -07:00
Chris Lu
133b772fb5 destroy ec volume if it is empty 2019-06-01 01:51:28 -07:00
Chris Lu
ba18314aab ec shard delete also check ec volumes, in addition to volumes 2019-06-01 01:41:22 -07:00
Chris Lu
f53024d79d delete ec collection 2019-05-30 10:40:22 -07:00
Chris Lu
1d111d6ce8 collection list normal and ec volumes 2019-05-30 09:27:23 -07:00
Chris Lu
e941d0c2f4 list ec collections also 2019-05-30 09:17:58 -07:00
Chris Lu
3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu
2858a954b3 read ec volume shards locations from master 2019-05-28 00:13:13 -07:00
Chris Lu
713596e781 caching ec shard locations 2019-05-27 22:54:58 -07:00
Chris Lu
217cde0a3b refactoring 2019-05-27 21:40:51 -07:00
Chris Lu
03b9291e5d volum server passes grpc option and master location to store 2019-05-27 21:22:23 -07:00
Chris Lu
ac1ba3b667 remove comments 2019-05-27 21:21:52 -07:00
Chris Lu
b4b407e403 add grpc ec shard read 2019-05-27 11:59:03 -07:00
Chris Lu
a4f3d82c57 convert needle id to ec intervals to read from 2019-05-27 01:29:46 -07:00
Chris Lu
92f9c7b1aa adjust ec shard status on disconnect 2019-05-26 01:05:08 -07:00
Chris Lu
b58e25e588 broadcast vid->location map to master clients for ec chards 2019-05-26 00:49:15 -07:00
Chris Lu
af67d99ca4 incrementally update master ec shards state 2019-05-26 00:21:17 -07:00
Chris Lu
db94a41f9e mount/unmount ec shards 2019-05-25 23:23:19 -07:00
Chris Lu
41e8ae61f8 generate, balance, delete copied shards, delete old volume 2019-05-25 14:02:06 -07:00
Chris Lu
6f4b09b6a4 pb shard info uses ShardBits instead one message for one shard 2019-05-24 13:28:44 -07:00
Chris Lu
24b8ad9a14 push the ec shard info to master 2019-05-23 23:47:49 -07:00
Chris Lu
4659d80035 prepare to register ec shard info in master 2019-05-23 00:42:28 -07:00
Chris Lu
17ac1290c0 volume: load ec shards during heartbeats to master 2019-05-21 22:41:20 -07:00
Chris Lu
fbbc74abb4 adds VolumeEcGenerateSlices, VolumeEcCopy 2019-05-20 00:53:17 -07:00
Chris Lu
698dea779c refactoring 2019-05-19 21:37:49 -07:00
Chris Lu
017d0957c3 avoid switching master node if same ip and host name 
fix https://github.com/chrislusf/seaweedfs/issues/955
 2019-05-11 18:42:46 -07:00
Chris Lu
8ffdd5dcdb writel throttling moving volumes 2019-05-06 13:58:42 -07:00
Chris Lu
3ff92037cc shell: add volume.balance command 2019-05-05 21:17:23 -07:00
Chris Lu
25941e0500 master: add jwt expires_after_seconds 2019-05-04 08:42:25 -07:00
Chris Lu
bd8af92b54 remove unused code 2019-05-04 08:20:22 -07:00
Chris Lu
5681208f2d refactor: remove logger 2019-05-03 22:21:06 -07:00
Chris Lu
b335f81a4f volume: add option to limit compaction speed 2019-05-03 17:22:39 -07:00
Chris Lu
f0f981e7c8 webdav: fix directory renaming 2019-05-03 14:12:51 -07:00
Chris Lu
bfbecd7253 webdav: fix directory creation 2019-05-03 13:13:11 -07:00
Chris Lu
9d3c945b00
Merge pull request #950 from chrislusf/webdav 
add WebDav
 2019-05-03 01:25:10 -07:00
Chris Lu
7064b2ea48 webdav: can write now 2019-05-03 00:55:52 -07:00
Chris Lu
b30c14b631 webdav: can read now 2019-05-03 00:24:35 -07:00
Chris Lu
47f14775d7 scaffolding, but seems hard to satisfy mac, windows, and different os versions 
https://www.reddit.com/r/golang/comments/3wrxo3/webdav_server_that_provide_filesystem_using/
 2019-05-02 14:22:10 -07:00
Wine93
32f93fb09e style: go fmt for all 2019-04-30 03:22:19 +00:00
Chris Lu
108d0fb08d adjust log level 2019-04-26 09:32:37 -07:00
Chris Lu
00b6f653fa shell: add fs.cat 2019-04-21 15:43:43 -07:00
Chris Lu
440111a349 volume: support http status 304 for the same file id 2019-04-21 13:33:23 -07:00
Chris Lu
0302b9496c volume: also check cookie for possible duplicates 2019-04-21 13:31:45 -07:00
Chris Lu
132921ad41 adjust log or error 2019-04-21 10:14:17 -07:00
Chris Lu
2ffe98443b refactoring 2019-04-20 12:05:28 -07:00
Chris Lu
6fc1f53018 shell: add command volume.move 2019-04-20 11:35:23 -07:00
Chris Lu
3b3651dea3 volume: atomic copying file, adds version and stopOffset 2019-04-19 12:29:49 -07:00
Chris Lu
ac2727853f fix needle map entry size 2019-04-19 00:39:34 -07:00
Chris Lu
e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu
fa176fe80f volume tailing chunks large files 2019-04-18 19:22:13 -07:00
Chris Lu
3dce1016cb add volume tailer 2019-04-18 11:05:02 -07:00
Chris Lu
b142f9f1d5 go fmt 2019-04-18 00:19:18 -07:00
Chris Lu
b09e8dbb37 add VolumeStreamFollow, but not used yet 2019-04-18 00:18:29 -07:00
Chris Lu
13ad5c1966 refactoring 2019-04-17 22:04:49 -07:00
Chris Lu
3b9446606a adjust log level 2019-04-16 23:31:44 -07:00
Chris Lu
338e6d60a5 refactor: prepare for snapshotting 2019-04-16 09:55:37 -07:00
Chris Lu
79c2cca9c1 better error message 2019-04-16 01:15:30 -07:00
Chris Lu
6e116b3d67 volume: reset leader if error happens 2019-04-15 09:09:46 -07:00
Chris Lu
e85048bcdc http exhaust and close response body 2019-04-14 23:28:24 -07:00
Chris Lu
3e8a3a8fec fix race detector found problems 2019-04-14 23:00:37 -07:00
Chris Lu
0e33272f56 small refactoring 2019-04-10 23:39:53 -07:00
Chris Lu
cb2061f86a
Merge pull request #925 from stlpmo-jn/checkUnhealthReplication 
the implement of repair the error replications of the volume
 2019-04-10 23:37:12 -07:00
Chris Lu
2f76681d62 refactor: adjust grpc API name 2019-04-10 21:41:17 -07:00
stlpmo-jn
f2031884f0 fix bug : CI build failed 2019-04-11 09:53:31 +08:00
stlpmo-jn
c1a0403da2 repair the error replications of the volume 2019-04-10 19:41:55 +08:00
Chris Lu
000ee725fc refactor Offset into a struct of bytes 2019-04-08 19:40:56 -07:00
Chris Lu
766396d249 weed master: atomic volume counting 
possible fix for https://github.com/chrislusf/seaweedfs/issues/913
 2019-04-04 19:27:00 -07:00
Chris Lu
189c890715 weed replicate: replicate atomic rename to other systems 2019-03-31 08:10:47 -07:00
Chris Lu
78ac2bef3c go fmt 2019-03-30 23:09:16 -07:00
Chris Lu
97406333a5 support atomic renaming for mysql/postgres filer store 2019-03-30 23:08:29 -07:00
Chris Lu
c7e7b6229f detect mime type 2019-03-27 14:25:18 -07:00
Chris Lu
65757ae2fd weed filer: set content-type consistent with filer store 2019-03-27 11:41:11 -07:00
Chris Lu
7a14cdc90c refactoring, go fmt 2019-03-25 23:18:40 -07:00
Chris Lu
19728fe3f6 remove deprecated code 2019-03-25 23:12:14 -07:00
Chris Lu
df95ce0b6c weed backup: efficient delta backup 
fix https://github.com/chrislusf/seaweedfs/issues/399
 2019-03-25 23:01:53 -07:00
Chris Lu
c913583e2d Merge branch 'master' into add_volume_follow 2019-03-25 21:19:34 -07:00
Chris Lu
a32797518b weed master: redirect to leader for default admin UI 
fix https://github.com/chrislusf/seaweedfs/issues/898
 2019-03-25 09:20:05 -07:00
Chris Lu
70815e9124 WIP 2019-03-25 09:16:12 -07:00
Chris Lu
95e0520182 weed volume: add grpc operation to relicate a volume to local 2019-03-23 11:33:34 -07:00
Chris Lu
a3490b600c weed filer, weed master: add option to disable http 2019-03-21 16:00:46 -07:00
Chris Lu
531add52c2 weed master: skip proxied file read in http read only mode 2019-03-21 09:49:04 -07:00
Jonathan Amsterdam
e966033098 Merge branch 'master' of https://github.com/chrislusf/seaweedfs into gocdk 2019-03-20 16:02:38 -04:00
Chris Lu
977b30e992 weed filer: http HEAD response header add ETag 
fix https://github.com/chrislusf/seaweedfs/issues/892
 2019-03-20 10:13:53 -07:00
Jonathan Amsterdam
8db82e2b75 notification: add Go CDK pubsub support 
Add the gocdk_pub_sub package, which supports the Go Cloud Development
Kit pubsub API.

Link in all current providers.

Update the notification scaffold.
 2019-03-20 07:57:58 -04:00
Chris Lu
88ab932f7d refactoring function parameter 2019-03-19 23:01:23 -07:00
Chris Lu
da871896c3 weed filer: set grpc port to port + 10000 2019-03-19 05:47:41 -07:00
Chris Lu
eb4a54d9fe refactoring 2019-03-19 05:19:37 -07:00
Chris Lu
104922a3db text wrapping 2019-03-18 00:35:15 -07:00
Chris Lu
aca653c08b weed shell: list volumes 2019-03-17 20:27:08 -07:00
Chris Lu
657dd2e6c9 add shell command to list all collections 2019-03-16 13:43:16 -07:00
Chris Lu
55bab1b456 add context.Context 2019-03-15 17:20:24 -07:00
Chris Lu
cece860bfd add context to all filer APIs 2019-03-15 15:55:34 -07:00
Chris Lu
55c85f3a66 adding context 2019-03-15 15:26:09 -07:00
Chris Lu
ad08a52ab6 synchronously upload files 
fix https://github.com/chrislusf/seaweedfs/issues/807
 2019-03-09 13:24:22 -08:00
Chris Lu
4773497d2c add back "/cluster/status" 
fix https://github.com/chrislusf/seaweedfs/issues/870
 2019-03-06 12:10:45 -08:00
Chris Lu
0898e6c38e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-03-04 13:00:10 -08:00
Chris Lu
d312c55bbe file path supports windows, avoiding back slashes 
fix https://github.com/chrislusf/seaweedfs/issues/868
 2019-03-04 13:00:08 -08:00
Chris Lu
f9dcf56465 adjust log level for loading configurations 2019-03-04 09:02:58 -08:00
chenwanli
f233bbe9c0 Fix https://github.com/chrislusf/seaweedfs/issues/825#issuecomment-469097375 2019-03-04 12:43:43 +08:00
Chris Lu
219b651bc3 jwt check the base file id 
fix https://github.com/chrislusf/seaweedfs/issues/867
 2019-03-03 10:17:44 -08:00
Chris Lu
2812c14520 master: add option to disable http operations 2019-03-02 05:10:05 -08:00
Chris Lu
e108688990 avoid grpc 5 seconds timeout 
some operations may take longer than 5 seconds.

only keep the timeout for raft operations
 2019-02-20 01:01:01 -08:00
Chris Lu
07af52cb6f raft change from http to grpc 
master grpc port is fixed to http port + 10000
 2019-02-18 22:38:14 -08:00
Chris Lu
d37c3ab7a5 adjust error message 2019-02-18 18:03:27 -08:00
Chris Lu
77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu
7103c1ab7e go fmt 2019-02-15 00:09:48 -08:00
Chris Lu
74fb237727 benchmark can work in secure mode 2019-02-15 00:09:19 -08:00
Chris Lu
215cd27b37 add authorizing fileId write access 
need to secure upload/update/delete for benchmark/filer/mount
need to add secure grpc
 2019-02-14 00:08:20 -08:00
Chris Lu
4ff4a147b2 cleanup security.Secret 2019-02-09 21:56:32 -08:00
Chris Lu
8afc632484 raft: use the first master to bootstrap the election 2019-02-09 12:52:09 -08:00
Chris Lu
744abc2690 log raft state changes 2019-02-09 12:51:14 -08:00
Chris Lu
1334507595 Revert "randomize based on self address" 
This reverts commit 6230eb28a6.
 2019-01-28 12:12:51 -08:00
Chris Lu
6230eb28a6 randomize based on self address 
fix #851
 2019-01-28 11:55:33 -08:00
Chris Lu
221105eea3 Revert "use the first entry to bootstrap master cluster" 
This reverts commit 40c8725ffa.
 2019-01-28 11:46:46 -08:00
Chris Lu
40c8725ffa use the first entry to bootstrap master cluster 
fix https://github.com/chrislusf/seaweedfs/issues/851
 2019-01-28 10:35:28 -08:00
Chris Lu
834f414af9 add a timeout 2019-01-26 00:15:42 -06:00
Chris Lu
3f56b12ed4 raft: adding idle connection time out 
another attempt to fix https://github.com/chrislusf/seaweedfs/issues/825
 2019-01-22 09:25:25 -08:00
Chris Lu
67e2ea72be master add separate grpc port 
due to https://github.com/soheilhy/cmux/issues/64

fix https://github.com/chrislusf/seaweedfs/issues/820
fix https://github.com/chrislusf/seaweedfs/issues/840
fix https://github.com/chrislusf/seaweedfs/issues/841
 2019-01-18 14:14:47 -08:00
Chris Lu
1d103e3ed5 timeout http connections 
possible fix for https://github.com/chrislusf/seaweedfs/issues/825
 2019-01-17 23:38:33 -08:00
Chris Lu
2ff95ead57
Merge pull request #832 from PapaYofen/fix-830 
Fix https://github.com/chrislusf/seaweedfs/issues/830
 2019-01-12 17:26:51 -08:00
chenwanli
b71d6f4b88 Fix https://github.com/chrislusf/seaweedfs/issues/830 2019-01-12 10:07:19 +08:00
chenwanli
80cf99db1b Set rpc dial timeout to 5s 2019-01-11 18:23:31 +08:00
chenwanli
0a3e83a36a Set timeout for master and volume non-streaming rpc 2019-01-10 19:41:03 +08:00
chenwanli
41fe31b5d0 Speed up heartbeat to new master leader 2019-01-07 16:09:15 +08:00
Chris Lu
7f74577db1 add Last-Modified to HEAD requests 
to work better with restic
 2019-01-02 14:23:25 -08:00
Chris Lu
28a41fda3e assign new file id even on filer PUT operations 2019-01-02 12:58:26 -08:00
Chris Lu
e49a38842a set etag in response 2019-01-02 12:57:54 -08:00
Chris Lu
0c932d1738 add volume delete grpc API 
fix https://github.com/chrislusf/seaweedfs/issues/808
 2018-12-29 00:03:30 -08:00
Chris Lu
260b038441 fix url 2018-12-27 14:43:36 -08:00
Chris Lu
be946c9e54 filer store original data size instead of data size after gzip 2018-12-22 13:11:07 -08:00
Chris Lu
5333f2984a streaming updates for large entries or large index file 
fix https://github.com/chrislusf/seaweedfs/issues/801
 2018-12-22 11:10:08 -08:00
Chris Lu
52712782a2 adding link 2018-12-17 00:31:23 -08:00
Chris Lu
2fb740a2b9 fix upload path 2018-12-17 00:21:34 -08:00
Chris Lu
7b7f4215aa filer update existing entry 2018-12-17 00:20:00 -08:00
Chris Lu
6071be104b adjust log format 2018-12-17 00:03:14 -08:00
Chris Lu
3ac54792e1 paginate when filer deleting and FUSE mount renaming 2018-12-16 23:20:08 -08:00
Chris Lu
93d0ad09af error log format 2018-12-16 17:37:10 -08:00
Chris Lu
52b24a9902 default "weed server -filer" to same directory as -mdir 2018-12-05 23:24:25 -08:00
Chris Lu
650d1af978 HCFS support change user name and group names 2018-12-04 00:46:00 -08:00
Chris Lu
7ace0efd65 revert prev 2018-12-02 23:25:37 -08:00
Chris Lu
738247d47e do not throw error if not found 2018-12-02 23:20:29 -08:00
Chris Lu
bea162c34c add warning for empty attributes 2018-12-02 22:57:59 -08:00
Chris Lu
7a6f49cd42 refactor a bit more 
fix https://github.com/chrislusf/seaweedfs/issues/777
 2018-11-29 00:07:54 -08:00
Chris Lu
86dd933596 go fmt 2018-11-23 00:26:15 -08:00
Chris Lu
444dfded84 add fs.FSStatfser for SeaweedFS weed mount 2018-11-23 00:24:51 -08:00
Chris Lu
e12b6c0306 add subimit count parameter 
related to https://github.com/chrislusf/seaweedfs/issues/631
 2018-11-22 13:45:29 -08:00
Chris Lu
0ed816d4e9 formatting error log 2018-11-22 00:42:56 -08:00
Chris Lu
5065d4ab2d master add grpc API for fileid assigning 2018-11-20 11:35:45 -08:00
Chris Lu
a4ceb051a7 use MD5 for ETag to be consistent with Amazon S3 2018-11-08 21:41:02 -08:00
Chris Lu
3674ad9f8e go fmt 2018-11-01 01:12:21 -07:00
Chris Lu
db584ff7f8 separate into notification.toml, add gcp pub/sub message queue 2018-11-01 01:11:09 -07:00
Chris Lu
4c97ff3717 support AWS SQS as file change notification message queue 2018-10-31 01:11:19 -07:00
zoe
a11c28cb4c
Update volume_server_handlers_write.go 
size declared and not used
 2018-10-31 09:36:32 +08:00
zoe
29eff0ecd7
Update volume_server_handlers_write.go 
返回实际文件大小，而不是needle的大小
 2018-10-29 21:05:51 +08:00
Chris Lu
6a756136ef go fmt 2018-10-23 23:59:49 -07:00
Chris Lu
46eb77f9bb move DiskStatus and MemStatus to protobuf 2018-10-15 22:25:28 -07:00
Chris Lu
eec951cad2 migrate volume sync to gRpc 2018-10-15 21:44:41 -07:00
Chris Lu
f8b2d3cacc move volume mount/unmount on volume server to grpc 2018-10-15 01:48:15 -07:00
Chris Lu
66a353dcb5 remove volume server /admin/volume/delete 2018-10-15 01:26:49 -07:00
Chris Lu
fda771c83f migrate volume sync status to grpc API on volume server 2018-10-15 01:19:15 -07:00
Chris Lu
b423bb9e2d migrate assign volume to grpc API on volume server 2018-10-15 00:40:46 -07:00
Chris Lu
f9410bcaa7 adjust log level for deletes 2018-10-15 00:39:56 -07:00
Chris Lu
8301519fb0 migrate delete collection to grpc API on volume server 2018-10-15 00:03:55 -07:00
Chris Lu
db152ca540 fix error log format 2018-10-14 23:50:54 -07:00
Chris Lu
67d1e572ca rename files 2018-10-14 23:47:29 -07:00
Chris Lu
b1daede91b move volume vacuum to gRpc 2018-10-14 23:12:43 -07:00
Chris Lu
91ac2e0dd9 go fmt 2018-10-14 00:30:20 -07:00
Chris Lu
ff66269b62 use grpc to replace http APIs for batch volume id lookup and batch delete 
1. remove batch volume id lookup http API /vol/lookup
2. remove batch delete http API /delete
 2018-10-14 00:12:28 -07:00
Chris Lu
41aba0c02c fix when no files in filer 2018-10-12 00:45:28 -07:00
Chris Lu
4af6b4eb0e fix master static resource display problem 2018-10-12 00:05:15 -07:00
Chris Lu
556382ff5f adding basic grpc to volume server 2018-10-11 01:16:33 -07:00
Chris Lu
dbe33236fb embed static resources via statik 2018-10-07 21:42:57 -07:00
Chris Lu
fffbc2d490 embed static resources via statik 2018-10-07 10:54:05 -07:00
Chris Lu
f11d531d45 adjust logging 2018-09-28 01:54:10 -07:00
Chris Lu
60c1ada4c1 remove filer grpc API GetEntryAttributes 2018-09-25 09:18:06 -07:00
Chris Lu
a60767c43a add more instructions 2018-09-23 09:26:39 -07:00
Chris Lu
7d6b2a4740 add ETag to upload results and chunks 2018-09-22 22:12:21 -07:00
Chris Lu
13d6a74eb2 consolidate file chunks during create 2018-09-22 22:11:37 -07:00
Chris Lu
788acdf527 add WIP filer.replicate 2018-09-17 00:27:56 -07:00
Chris Lu
d923ba2206 renaming msgqueue to notification 2018-09-16 01:18:30 -07:00
Chris Lu
bea4f6ca14 go fmt 2018-09-12 01:00:57 -07:00
Chris Lu
f6d8525d1d working S3 multipart uploads 2018-09-12 00:46:12 -07:00
Chris Lu
164091c269 add s3 multipart upload 2018-09-09 16:25:43 -07:00
Chris Lu
d7a4b12d2a fix build error 2018-09-07 14:21:10 -07:00
Chris Lu
75e749039b fix data writes error when consecutive calls overlaps with previous writes 2018-09-07 13:11:43 -07:00
zhangmingfeng
aafe8d84cb
Update volume_server_handlers_read.go 
change ext to lower-case
 2018-09-03 11:01:35 +08:00
Chris Lu
76cbe8bf33 instant notification of new volumes added or deleted 2018-08-24 01:26:56 -07:00
Chris Lu
5ccf8e8078 reformat 2018-08-24 00:30:03 -07:00
Chris Lu
98b8f8649d adding filer.export prototype 2018-08-22 23:54:22 -07:00
Chris Lu
bb92fd1db0 fix test error 2018-08-19 18:52:50 -07:00
Chris Lu
906472b21d filer add upload and drag&dorp 2018-08-19 18:42:40 -07:00
Chris Lu
95b997fb35 go fmt 2018-08-19 15:18:37 -07:00
Chris Lu
f827ada811 merge notification config with filer.toml 2018-08-19 15:17:55 -07:00
Chris Lu
5c467083e5 go fmt 2018-08-13 01:22:32 -07:00
Chris Lu
f036ef8a3c add filer notification 2018-08-13 01:20:49 -07:00
Chris Lu
75d63db60d randomize raft server startup 
also some go fmt
 2018-08-12 14:27:14 -07:00
Chris Lu
b931ced0a3 readable file size 2018-08-10 23:47:31 -07:00
Chris Lu
032f1226db fix compilation error 2018-07-29 02:25:03 -07:00
Chris Lu
d3205a0070 go fmt 2018-07-28 21:02:56 -07:00
Chris Lu
888eb2abb5 filer read write all via locations from MasterClient 2018-07-28 14:51:36 -07:00
Chris Lu
1d779389cb MasterClient replicates all vid locations 2018-07-28 14:22:46 -07:00
Chris Lu
1ab8232b55 filer only talk to the master leader 2018-07-28 01:30:03 -07:00
Chris Lu
e8d4be579d send initial list of volume location 2018-07-28 01:17:35 -07:00
Chris Lu
a12c7b86b0 broadcast messages of new and deleted volumes 2018-07-27 23:09:55 -07:00
Chris Lu
e4b7e31902 add back recursively delete a folder 2018-07-24 22:33:26 -07:00
Chris Lu
4f317c7e3d working version3 2018-07-24 01:36:04 -07:00
Chris Lu
0b99557c29 fix compilation error 2018-07-24 01:09:32 -07:00
Chris Lu
9bf93e698e refactoring: avoid common code 2018-07-24 00:58:52 -07:00
Chris Lu
aae77e8ea3 initialize read seeker 2018-07-24 00:54:15 -07:00
Chris Lu
5a237af973 fix nil if not images 2018-07-24 00:49:53 -07:00
Chris Lu
ff331178aa
Merge pull request #695 from zhangmingfeng/master 
增加chunk图片文件支持width和height
 2018-07-22 19:54:04 -07:00
Chris Lu
7e76a7d21b
Merge pull request #694 from himyxia/inprove_switch_case 
better way to handler switch case
 2018-07-22 10:29:37 -07:00
Jimmy
1c77d323b0 better way to handler switch case 2018-07-23 01:27:10 +08:00
Chris Lu
056871242b remove println 2018-07-22 01:17:49 -07:00