mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-27 19:37:55 +08:00
Code Issues Packages Projects Releases Wiki Activity
5,657 Commits 32 Branches 296 Tags 153 MiB
aabea37af1
Commit Graph

3622 Commits

Author SHA1 Message Date
Chris Lu
3fbf635308 Merge branch 'master' into msg_channel 2020-05-13 03:54:15 -07:00
Chris Lu
3a5e4769cf prevent empty locations returned 
fix https://github.com/chrislusf/seaweedfs/issues/1313
 2020-05-13 03:46:38 -07:00
Chris Lu
ca4017dd87 rename 2020-05-12 22:55:55 -07:00
Chris Lu
25257acd51 rename 2020-05-12 21:26:49 -07:00
Chris Lu
a7959c1c48 multiple subscriber with same subscriberId shares the topic manager 
rename topicControl to topicCursor
 2020-05-12 21:26:02 -07:00
Chris Lu
2f243f5b0b refactor 2020-05-12 08:48:00 -07:00
Chris Lu
4b7fa31468 ensure montonically increasing tsNs 2020-05-11 01:53:54 -07:00
Chris Lu
d5a8297a1c able to connect publisher 2020-05-10 23:45:48 -07:00
Chris Lu
d58371c5cd Merge branch 'master' into msg_channel 2020-05-10 23:44:59 -07:00
Chris Lu
015dd3a147 batch file id deletion 2020-05-10 23:25:39 -07:00
Chris Lu
39e72fb23c go fmt 2020-05-10 03:50:30 -07:00
Chris Lu
6bf3eb69cb async chan write read, no write for closed chan 2020-05-10 03:48:35 -07:00
Chris Lu
78afb8bf46 Merge branch 'master' into msg_channel 2020-05-09 00:43:58 -07:00
Chris Lu
d693e77418 add pub sub md5 2020-05-09 00:43:53 -07:00
Chris Lu
07d7abe428 add deleteTopic, refactoring 2020-05-09 00:31:34 -07:00
Chris Lu
dfccc3c263 able to read chan and write chan 2020-05-08 02:47:22 -07:00
zhangsong
b161baa3f0 fix volume vacuum test unit 2020-05-07 19:58:06 +08:00
Chris Lu
a8bc8eb351 Merge branch 'master' into msg_channel 2020-05-07 03:18:24 -07:00
Chris Lu
73087f59b1 filer store: mongodb deleting children 
fix https://github.com/chrislusf/seaweedfs/issues/1303
 2020-05-07 03:11:26 -07:00
Chris Lu
be1d9f324f Merge branch 'master' into msg_channel 2020-05-07 01:43:34 -07:00
Chris Lu
0952eb2854
Merge pull request #1302 from song-zhang/master 
write request keep compatible with before
 2020-05-06 15:37:34 -07:00
Chris Lu
5568395edd Revert "Revert "Merge pull request #1299 from song-zhang/master"" 
This reverts commit afb6a1dbb4.
 2020-05-06 15:37:17 -07:00
zhangsong
3703c7e176 write request keep compatible with before 2020-05-06 21:35:32 +08:00
Chris Lu
1e3e4b3072 add broker connects to filer 2020-05-05 02:05:28 -07:00
Chris Lu
afb6a1dbb4 Revert "Merge pull request #1299 from song-zhang/master" 
This reverts commit 9016fa19ba, reversing
changes made to 47234760f4.
 2020-05-04 20:34:26 -07:00
Chris Lu
9016fa19ba
Merge pull request #1299 from song-zhang/master 
use async write to persistent file to disk - part1
 2020-05-04 15:51:03 -07:00
zhangsong
f9e8702bb4 use async write to persistent file to disk - part1 2020-05-04 17:39:44 +08:00
Chris Lu
47234760f4 log adjust format 2020-05-02 01:33:49 -07:00
Chris Lu
97a504b0bb fix possible directory deletion error 2020-05-02 01:11:53 -07:00
Chris Lu
78422c2f69
Merge pull request #1298 from song-zhang/master 
add fsync option for benchmark
 2020-04-30 20:16:42 -07:00
Chris Lu
871efa4fc1 refactoring 
some previous chunk etag was using md5, which should be wrong.
 2020-04-30 17:20:47 -07:00
zhangsong
d819865fe7 add fsync option for benchmark 2020-04-30 23:31:08 +08:00
Chris Lu
ea809a45ab add cipher key 2020-04-30 03:05:43 -07:00
Chris Lu
f9b6178b8f log messages 2020-04-30 03:05:34 -07:00
Chris Lu
8c73410a51 subscribe from a timestamp 2020-04-30 02:19:51 -07:00
Chris Lu
4e16a90454 for debugging 2020-04-30 02:19:32 -07:00
Chris Lu
a25a8d8822 s3: print out time in UTC format 
fix https://github.com/chrislusf/seaweedfs/issues/1297
 2020-04-29 21:35:24 -07:00
Chris Lu
e93588ec78 FUSE mount: atomic local cache updates 2020-04-29 18:20:54 -07:00
Chris Lu
9e72e9e4b8 able to subscribe any topic from any point of time 2020-04-29 17:40:08 -07:00
Chris Lu
ed3cf811f5 refactoring 2020-04-29 13:26:02 -07:00
Chris Lu
7c10602b49 read in case cross chunks 2020-04-29 02:42:58 -07:00
Chris Lu
f9da859720 better error message 2020-04-29 02:42:06 -07:00
Chris Lu
d57559d3f7 chunk rememober encryption or not 2020-04-29 02:41:47 -07:00
Chris Lu
5a0986dca0 reset on getting the buffer 2020-04-29 02:41:01 -07:00
Chris Lu
8d42a1263b meta subscription: update the last read time 2020-04-28 17:30:04 -07:00
Chris Lu
31583b5d6c master and volume server: avoid race condition 
The volume server may disconnect and reconnect to the same master.

The master's unregistration may happen after the reconnection. Thus the volume server will disappear.
 2020-04-28 17:29:10 -07:00
Chris Lu
98edec7c3e remove printouts 2020-04-28 16:45:48 -07:00
Chris Lu
8ed490164e refactoring 2020-04-28 02:05:44 -07:00
Chris Lu
66effaed9e fix compilation 2020-04-28 02:05:32 -07:00
Chris Lu
eab4c9219c randomize benchmark content 
fix https://github.com/chrislusf/seaweedfs/issues/1294
 2020-04-28 01:50:56 -07:00
Chris Lu
fb81f12686 filer: simplify image resize 2020-04-28 00:05:57 -07:00
Chris Lu
5c57297bd1 metadata log: read from any timestamp 2020-04-28 00:05:57 -07:00
wuyuxiang
6850d28d6b refacotr: move signal handling and pprof to grace package 2020-04-28 14:10:23 +08:00
Chris Lu
2a7957b4ca FUSE: subcribe meta events based on timestamp 2020-04-27 16:52:20 -07:00
Chris Lu
6190fd665d printout error 2020-04-27 12:41:31 -07:00
Chris Lu
725e47dcc3 filer: pass along user name and group name for HDFS 2020-04-27 05:48:48 -07:00
Chris Lu
508ec01ab0 tweaking 2020-04-26 16:11:06 -07:00
Chris Lu
f4fcde05b6 1.77 2020-04-26 05:54:47 -07:00
Chris Lu
b52b8ec685 Hadoop: fix entry not found for HCFS 
also fix cipher related changes.
 2020-04-26 05:21:54 -07:00
Chris Lu
0c2248f83a add lock/unlock to example admin scripts 2020-04-23 18:41:48 -07:00
Chris Lu
913aa0f14e mount: default to enable asyncMetaDataCaching 2020-04-23 14:14:28 -07:00
Chris Lu
662b5d0cf7 support semicolon seperated command lines 2020-04-23 14:01:46 -07:00
Chris Lu
73564e6a01 master: add cluster wide lock/unlock operation in weed shell 
fix https://github.com/chrislusf/seaweedfs/issues/1286
 2020-04-23 13:37:31 -07:00
Chris Lu
bdc337a719 make lock/unlock optional 2020-04-23 03:50:05 -07:00
Chris Lu
57ec7f2ba9 fix builds 2020-04-23 03:43:45 -07:00
Chris Lu
eab9701823 allow lock with an existing lock 2020-04-23 03:35:52 -07:00
Chris Lu
842e3301ee make second lock instant 2020-04-23 03:32:18 -07:00
Chris Lu
69f336e59f shell: add lock/unlock command 2020-04-23 03:11:07 -07:00
Chris Lu
ff0a7c1d18 clean up 2020-04-23 03:10:47 -07:00
Chris Lu
30ee4f3291 add exclusive lock library on shell 2020-04-23 02:31:04 -07:00
Chris Lu
77873b832b add master side code for cluster wise exclusive lock 2020-04-23 01:55:44 -07:00
Chris Lu
369aa8a10a 1.76 2020-04-22 23:31:59 -07:00
Chris Lu
69eda558ef reduce the connections 
fix https://github.com/chrislusf/seaweedfs/issues/1277

this avoids sending error to wdclients, and clients retry the connections
 2020-04-22 20:43:34 -07:00
Chris Lu
aebe39a803 avoid repeated grpc connection creation 
fix https://github.com/chrislusf/seaweedfs/issues/1277
 2020-04-22 18:26:24 -07:00
Chris Lu
5d0e1d8d74 also writes to local meta cache 
before waiting for subscribed meta events
 2020-04-22 15:40:47 -07:00
Chris Lu
72c5dba4d4 1.75 2020-04-21 22:03:07 -07:00
Chris Lu
a207285af7 cache metadata on startup 2020-04-21 22:00:34 -07:00
Chris Lu
9f9826e95a refactor 2020-04-21 21:28:47 -07:00
Chris Lu
e24b25de78 async meta caching: can stream updates now 2020-04-21 21:16:13 -07:00
Chris Lu
4f02f7121d read from meta cache 
meta cache is not initialized
 2020-04-21 18:50:30 -07:00
Chris Lu
b8e4238ad2 add placeholder for cached meta data reading 2020-04-21 18:02:08 -07:00
Chris Lu
27128c7875 adjust logging 2020-04-21 16:13:13 -07:00
Chris Lu
bafa95045b volume: deletion checks all disk locations 
fix https://github.com/chrislusf/seaweedfs/issues/1283
 2020-04-21 14:49:58 -07:00
Chris Lu
57df14f76f report error if copying failed 2020-04-21 14:28:46 -07:00
Chris Lu
cf54d70311 filer: add -ip.bind option 2020-04-21 14:21:06 -07:00
Chris Lu
5c348087dc messaging: able to pub sub multiple partitions 2020-04-21 00:59:55 -07:00
Chris Lu
cb3985be70 go fmt 2020-04-20 17:48:06 -07:00
Chris Lu
5eb83dfbd3 add memory buffer size limit 2020-04-20 17:43:50 -07:00
Chris Lu
258fba8a0f continue for reading from sealed memory buffer 2020-04-20 17:28:18 -07:00
Chris Lu
4bf959edf0 message broker: read also from sealed memory buffer 2020-04-20 17:26:38 -07:00
Chris Lu
bd43c62fbd lower the test criteria 2020-04-20 03:08:10 -07:00
Chris Lu
c3f9fdf48f adjust test 2020-04-20 02:54:21 -07:00
Chris Lu
5e8bb86995 close the connection with better accouting 2020-04-20 02:53:50 -07:00
Chris Lu
978da73cdb revert this grpc related change 
maybe related to https://github.com/chrislusf/seaweedfs/issues/1277
 2020-04-20 02:53:22 -07:00
Chris Lu
ebfab42a50 refactoring 2020-04-20 00:08:47 -07:00
Chris Lu
2955b96ef1 refactor 2020-04-19 23:54:32 -07:00
Chris Lu
df25ed3c5b logs 2020-04-19 23:37:50 -07:00
Chris Lu
ce3cb25cfb working for in memory single log buffer 2020-04-19 23:37:50 -07:00
Chris Lu
f373232227
Merge pull request #1280 from bukton123/filer_mongodb 
Add new filer support mongodb
 2020-04-19 20:35:19 -07:00
bukton
da37455254 filer mongo add func create index unique directory, name 2020-04-20 08:13:57 +07:00
Chris Lu
4642ed565e fix tests 2020-04-19 10:42:58 -07:00
bukton
3defa718a1 add default config filer mongodb to scaffold.go 2020-04-19 17:52:38 +07:00
bukton
d6d245dc89 filer mongodb add option poll size and fixed order by list meta 2020-04-19 17:51:32 +07:00
Chris Lu
71ffb98475 broker: add profiling 2020-04-19 03:03:40 -07:00
Chris Lu
d278b4c228 basic pub and sub works! 2020-04-19 00:18:32 -07:00
bukton
082f079ead filer mongo change func update to insert entry 2020-04-19 11:08:10 +07:00
Chris Lu
788578d4ea add subscriber api 2020-04-18 16:05:29 -07:00
Chris Lu
767f14dfcd master, volume default to detected host 2020-04-18 15:20:20 -07:00
Chris Lu
076c8bd3bc filer master start up with default ip address instead of just localhost 2020-04-18 15:17:27 -07:00
Chris Lu
5d346d44bd Revert "move go client out" 
This reverts commit ef1274edc9.
 2020-04-18 12:44:55 -07:00
Chris Lu
ef1274edc9 move go client out 2020-04-18 12:11:50 -07:00
Chris Lu
5af49c7aa5 adjust API 2020-04-18 11:45:57 -07:00
bukton
8c58b0d017 filer mongodb delete folder children 2020-04-19 00:31:49 +07:00
bukton
290c6b7f01 Merge remote-tracking branch 'origin/master' into filer_mongodb 
# Conflicts:
#	go.mod
#	go.sum
#	weed/server/filer_server.go
 2020-04-19 00:21:45 +07:00
bukton
6234ea441b filer mongodb delete 2020-04-19 00:20:00 +07:00
bukton
24a8192298 filer mongo list 2020-04-19 00:16:50 +07:00
bukton
3c70163798 filer mongodb insert entry and find entry 2020-04-18 23:48:38 +07:00
bukton
cd9cccec55 add import mongodb to filer server 2020-04-18 22:30:13 +07:00
bukton
fd4576651b add new package mongodb and install lib mongodb 2020-04-18 22:26:57 +07:00
Chris Lu
11f5a6d913 messaging can compile now 2020-04-18 01:12:01 -07:00
Chris Lu
5bea77010f refactor 2020-04-17 22:39:21 -07:00
Chris Lu
2e2537a9ea rename files 2020-04-17 22:35:44 -07:00
Chris Lu
e0f5996560 fix "call of Unmarshal passes non-pointer as second argument" 2020-04-17 10:04:41 -07:00
Chris Lu
826f96b6f3 fix unreachable code 2020-04-17 10:04:28 -07:00
Chris Lu
ffa69b24ae fix travis 2020-04-17 10:01:11 -07:00
Chris Lu
3f3dba5a68 broker: append message logs 2020-04-17 02:29:38 -07:00
Chris Lu
2a45897237 broker: read cipher value from filer 2020-04-17 02:29:00 -07:00
Chris Lu
bda82f61bc filer: able to append to a file 2020-04-17 02:28:09 -07:00
Chris Lu
722b14337e add cipher option to meta data updates 2020-04-17 01:26:27 -07:00
Chris Lu
7d3672c60a simplify 2020-04-17 00:00:48 -07:00
Chris Lu
508f3490a0 update messaging proto 2020-04-16 03:29:57 -07:00
Chris Lu
f5a748d33c refactoring 2020-04-16 02:55:09 -07:00
Chris Lu
ce4b369be2 scaffolding messaging 2020-04-16 02:21:23 -07:00
Reed
aada5b9aa7 making volume stats size info readable. 2020-04-16 10:08:08 +08:00
Chris Lu
9cacaf5eca better detect file mime type 2020-04-14 11:32:31 -07:00
Chris Lu
3f1d79512f only detect when not sure about the types 2020-04-14 11:02:05 -07:00
Chris Lu
d2d1597850 volume: detect gzip effectiveness only when content type is empty 
revert part of 2286eda575
 2020-04-14 10:52:53 -07:00
Chris Lu
23715af402
Merge pull request #1273 from wyxloading/config_race 
Fix data race about config
 2020-04-13 22:24:51 -07:00
Chris Lu
2b5c4fbbf3 tiered caching 
1/4 for small less than 1MB files. 1/4 for 1~4MB files, 1/2 for bigger than 4MB files
 2020-04-13 22:19:27 -07:00
Chris Lu
f282ed444b refactoring 2020-04-13 21:58:10 -07:00
wuyuxiang
b556d3d035 Fix data race about config 2020-04-14 10:22:40 +08:00
Chris Lu
d8f5985e5e add logs 2020-04-13 12:58:45 -07:00
Chris Lu
5c9e91393f 1.74 2020-04-13 10:25:48 -07:00
Chris Lu
7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu
8e23dc078b refactoring 2020-04-12 20:48:21 -07:00
Chris Lu
b062393f3f refactoring 2020-04-12 17:51:31 -07:00
Chris Lu
5361f999ed shell: only one shell is allowed to connect to the cluster 
fix https://github.com/chrislusf/seaweedfs/issues/1266
 2020-04-12 17:48:39 -07:00
Chris Lu
eb16bb0287 go mod cleanup 2020-04-12 16:44:58 -07:00
Chris Lu
d30483d642 re-enable system logs 2020-04-12 14:03:07 -07:00
Chris Lu
6f948e4887 remove configurable topics folder location 2020-04-12 13:07:59 -07:00
Chris Lu
94e35cdb35 mount: fix 
fix bug found by git bisect, but I do not understand why it can cause error!
 2020-04-12 03:34:36 -07:00
Chris Lu
1c2e920fa2 fix compilation error 2020-04-12 03:11:19 -07:00
Chris Lu
b5a713be68 filer store: add redis2 redis_cluster2 as default redis store 2020-04-12 02:50:41 -07:00
Chris Lu
e613695af7 a little optimization 2020-04-12 01:13:57 -07:00
Chris Lu
2a1f396df5 avoid duplicated setting chunks into cache 2020-04-12 01:06:50 -07:00
Chris Lu
b9b7da905e handle nil chunk cache 2020-04-12 01:00:12 -07:00
Chris Lu
211d87cf4c mount: option to disable caching 2020-04-12 00:52:54 -07:00
Chris Lu
1c65656fb4 s3: add option to fsync buckets 2020-04-11 23:37:10 -07:00
Chris Lu
e4af63a721 volume server: accept fsync=true in write requests 2020-04-11 21:39:16 -07:00
Chris Lu
5987810e5e set default chunk size to 16 2020-04-11 21:22:52 -07:00
Chris Lu
df97da25f9 mount: add on disk caching 2020-04-11 21:12:41 -07:00
Chris Lu
c8ca234773 refactoring 2020-04-11 14:27:25 -07:00
Chris Lu
df9d538044 rename function 2020-04-11 14:19:44 -07:00
Chris Lu
d7f3acb2c0 refactor 2020-04-11 12:45:24 -07:00
Chris Lu
ed54682598 refactoring 2020-04-11 12:43:17 -07:00
Chris Lu
e909b55633 refactoring 2020-04-11 12:39:46 -07:00
Chris Lu
b7f0ba3800 refactoring 2020-04-11 12:37:41 -07:00
Chris Lu
417125457e add additional buffers for logs 2020-04-11 02:19:48 -07:00
Chris Lu
4b27afe176 fix sqs message reading 2020-04-10 15:27:47 -07:00
Chris Lu
dce2702e1b fix the instruction 2020-04-10 14:50:10 -07:00
Chris Lu
13ab1fdaf0 mount: use file default permission 0666 2020-04-10 02:23:23 -07:00
Chris Lu
1101a42e5c mv : create new folder, move children, and delete old folder 2020-04-10 01:37:03 -07:00
Chris Lu
bcf37346ef add timestamp inside lock 2020-04-10 01:35:59 -07:00
Chris Lu
9fa065f600 typo in logs 2020-04-09 23:43:09 -07:00
Chris Lu
8db2120bee add logs 2020-04-09 23:42:59 -07:00
Chris Lu
43c62a8a95 1.73 2020-04-09 21:44:30 -07:00
Chris Lu
eb39df2704 avoid dead loop 2020-04-09 21:43:26 -07:00
Chris Lu
56ec8c4513 fix to avoid nil file an dirty pages 2020-04-09 21:43:05 -07:00
Chris Lu
59f40e2027 volume: best effort to detect ip address 
fix https://github.com/chrislusf/seaweedfs/issues/1264
 2020-04-09 00:26:24 -07:00
Chris Lu
f6a7e79dc3 weed shell: simplify CLI option for filer 2020-04-08 23:57:15 -07:00
Chris Lu
2329d9e0c1 add volume.fix.replication to default scaffold 2020-04-08 23:36:22 -07:00
Chris Lu
bb78ab9915 wait for master to be aware of the failed volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1268
 2020-04-08 23:12:37 -07:00
Chris Lu
8764bdb9df 1.72 2020-04-08 22:35:14 -07:00
Chris Lu
8a73276602 refactor a bit 2020-04-08 22:33:36 -07:00
Chris Lu
006f78b70b make a copy of request data 
to avoid concurrency issues
 2020-04-08 22:31:19 -07:00
Chris Lu
00b993a234 add util for md5 2020-04-08 22:30:38 -07:00
Chris Lu
ed98223b08 still log, but not persisting the changes 2020-04-08 20:32:57 -07:00
Chris Lu
6630541399 ensure correct file size when opening existing files 2020-04-08 12:50:59 -07:00
Chris Lu
b524a40375 add locking to fs cache 2020-04-08 12:50:34 -07:00
Chris Lu
bd56172b82 simplify file handle management 2020-04-08 12:50:20 -07:00
Chris Lu
dc08e4098f add etag only for PUT or large chunked uploads 2020-04-08 09:13:26 -07:00
Chris Lu
ec2eb8bc48 add If-None-Match and If-Modified-Since 
fix https://github.com/chrislusf/seaweedfs/issues/1269
 2020-04-08 08:12:00 -07:00
Chris Lu
4c498e73de filer replication: add s3 endpoint 2020-04-07 17:49:00 -07:00
Chris Lu
745f5d2a25 fix setting default value 2020-04-07 01:58:48 -07:00
Chris Lu
54debdc6f7 filer: fix configuration settings 2020-04-07 01:30:53 -07:00
Chris Lu
90d576affe disable meta data change event logging for now. 2020-04-07 01:05:24 -07:00
Chris Lu
b51d8f03bb 1.71 2020-04-05 22:04:16 -07:00
Chris Lu
5a5908407d filer: support larger file size 
fix https://github.com/chrislusf/seaweedfs/issues/1257
 2020-04-05 21:48:45 -07:00
Chris Lu
19edd9c091 comment out debug messages 2020-04-05 20:15:17 -07:00
Chris Lu
78ea937606 fix 2020-04-05 17:34:26 -07:00
Chris Lu
53626734d4 iterate through the log buffer 2020-04-05 16:51:30 -07:00
Chris Lu
f71b855db5 adjust caching and notification for recursive directory deletion 2020-04-05 15:03:25 -07:00
Chris Lu
f14b6a09a8 remove duplicated update events 2020-04-05 13:22:15 -07:00
Chris Lu
91da7057b1 refactoring 2020-04-05 13:11:43 -07:00
Chris Lu
2d43f85577 watch entries with common path prefix 2020-04-05 12:51:21 -07:00
Chris Lu
7bc3c93512 add util.PathJoin 2020-04-05 12:40:46 -07:00
Chris Lu
af1f64d244 change from 'weed tail' to 'weed watch' 2020-04-05 12:39:20 -07:00
Chris Lu
b10679fcf0 adjust log file names 2020-04-05 12:38:46 -07:00
Chris Lu
2cb1700ecb set to 4MB meta log buffer size 2020-04-05 12:38:29 -07:00
Chris Lu
e78ee087e9 fix tests 2020-04-05 01:17:35 -07:00
Chris Lu
dcc6b6a3ed go fmt 2020-04-05 00:52:31 -07:00
Chris Lu
bf270d9e8c filer: able to tail meta data changes 2020-04-05 00:51:16 -07:00
Chris Lu
f7b5a1d697 add disk stat to logs 2020-04-04 13:17:23 -07:00
Chris Lu
6e776aaafc loop through all masters 2020-04-03 00:47:48 -07:00
Chris Lu
4a5e381feb fail fast if filer configured wrong masters 
fix https://github.com/chrislusf/seaweedfs/issues/1257
 2020-04-03 00:47:33 -07:00
Chris Lu
aaf5e658cc adjust parameter name 2020-04-03 00:41:05 -07:00
Chris Lu
6c96e51414 re-ordering 2020-04-03 00:40:54 -07:00
Chris Lu
d2f8e50459 make error a little bit more clearer 
fix https://github.com/chrislusf/seaweedfs/issues/1259
 2020-04-03 00:24:09 -07:00
Chris Lu
5f881d2fd5 mount: fail early rather than after mounted the directory 
fix https://github.com/chrislusf/seaweedfs/issues/1258
 2020-04-03 00:21:03 -07:00
Chris Lu
fee3319cbf handle gzip for image resizing 2020-04-02 23:44:24 -07:00
Chris Lu
73bc286377 volume: passing ext info via URL 2020-04-02 21:23:15 -07:00
Chris Lu
cea52a4faf volume copying adds cleaning up on error 
fix https://github.com/chrislusf/seaweedfs/issues/1253
 2020-04-02 15:36:55 -07:00
Chris Lu
c981975627 volume.fix.replication: add test cases for complicated moving 
fix https://github.com/chrislusf/seaweedfs/issues/1253
 2020-04-02 02:16:16 -07:00
Chris Lu
ae3d08883f volume.fsck: fix for erasure encoded volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1254
 2020-04-01 22:10:13 -07:00
Chris Lu
c446438ca5
Merge pull request #1255 from levenlabs/ignore 
Added treat_replication_as_minimums master toml option
 2020-04-01 16:48:58 -07:00
James Hartig
eae3f27c80 Added treat_replication_as_minimums master toml option 2020-04-01 19:08:48 -04:00
Chris Lu
2fc2eb74dd refactoring 2020-03-30 22:57:45 -07:00
Chris Lu
c22094d340 adjust logging 2020-03-30 13:03:43 -07:00
Chris Lu
50a5018b7f writing meta logs is working 2020-03-30 01:19:33 -07:00
Chris Lu
9dc0b1df8f refactoring to compile for windows OS 
windows os does not like to work with fuse
 2020-03-29 21:07:55 -07:00
Chris Lu
be6c2f8511 1.70 2020-03-29 13:40:55 -07:00
Chris Lu
54768d0761 fix: delete a file and then create a directory with the same name 2020-03-29 01:39:48 -07:00
Chris Lu
057722bbf4 return part of the chunk if chunkview is not the full chunk 2020-03-29 00:54:39 -07:00
Chris Lu
af6d9976ad adjust print out 2020-03-28 14:07:25 -07:00
Chris Lu
a75d50bbb8 FUSE: add configurable in memory chunk cache size 2020-03-28 14:07:16 -07:00
Chris Lu
826bc0b7e3 FUSE: add chunk cache for recently accessed file chunks 2020-03-28 13:43:31 -07:00
Chris Lu
4aa82c95e6 refactor 2020-03-28 13:42:35 -07:00
Chris Lu
eedd33dda3 refactoring 2020-03-28 13:41:58 -07:00
Chris Lu
005b4ab3fe 1.69 2020-03-27 04:51:31 -07:00
Chris Lu
d1439c5bd3 fix FUSE read for large files 
FUSE expects ReadAt do not return partial filled buffer with a nil error.
 2020-03-27 04:50:51 -07:00
Chris Lu
f06ca04451 avoid overflow 2020-03-27 04:35:31 -07:00
Chris Lu
e1911760a7 refactoring 2020-03-27 00:30:55 -07:00
Chris Lu
2f5ed29c37 clean up 2020-03-27 00:30:39 -07:00
Chris Lu
2286eda575 sampling whether the data can be gzipped 2020-03-26 23:50:48 -07:00
Chris Lu
5463560912 small test 2020-03-26 23:50:07 -07:00
Chris Lu
34a0d424f8 FUSE mount: report max volume count available disk space 2020-03-26 23:06:21 -07:00
Chris Lu
a5ce359999 refactoring 2020-03-26 22:46:49 -07:00
Chris Lu
0a05052b5c 1.68 2020-03-26 22:22:36 -07:00
Chris Lu
e7e8691720 add some tests 2020-03-26 22:21:52 -07:00
Chris Lu
86398d1efe S3 API: set deletion to recursive 
fix https://github.com/chrislusf/seaweedfs/issues/1251
 2020-03-26 11:05:54 -07:00
Chris Lu
1fef598347 set target Dir object when moving 2020-03-26 10:56:40 -07:00
Chris Lu
9e2e07b238 tweak 2020-03-26 10:56:18 -07:00
Chris Lu
cbce793753 rename: set new name for renamed files 2020-03-26 03:30:23 -07:00
Chris Lu
c763a492f5 refactor 2020-03-26 03:30:02 -07:00
Chris Lu
712720864d error checking for remove 2020-03-26 02:21:30 -07:00
Chris Lu
8d9a76005d add back running output 2020-03-26 02:14:48 -07:00
Chris Lu
dd5b582d05 go fmt 2020-03-26 00:09:01 -07:00
Chris Lu
7f0d87b206 tree structured fs cache 
FsCache for FsNode, wrapping fs.Node
 2020-03-26 00:08:14 -07:00
Chris Lu
2e4fadd10a fix fscache move 2020-03-25 22:19:19 -07:00
Chris Lu
966df2ae27 purging skip EC volumes 2020-03-25 02:41:22 -07:00
Chris Lu
6d87a0c5f5 batch delete orphan data by file keys 2020-03-25 02:21:15 -07:00
Chris Lu
5ebc95b69b refactoring 2020-03-25 02:20:19 -07:00
Chris Lu
d79f0a76b1 format output 2020-03-25 00:56:47 -07:00
Chris Lu
d9b56ed88b shell: volume.fsck add options 2020-03-24 22:38:36 -07:00
Chris Lu
e63a79ade8 better handle lock in case of exception 2020-03-24 18:41:25 -07:00
Chris Lu
4d5554b16f fix compilation on travis 2020-03-24 18:40:52 -07:00
Chris Lu
d83e648c0b auto add prefix "fs." 2020-03-24 02:40:51 -07:00
Chris Lu
a875f67a08 fix pwd printout 2020-03-24 02:40:29 -07:00
Chris Lu
00d3f01876 better output format 2020-03-24 02:34:28 -07:00
Chris Lu
0820935290 clean up 2020-03-24 02:20:37 -07:00
Chris Lu
49440cbd2f shell: add volume.fsck 
volume.fsck can find out how much space is not used by the filer
 2020-03-24 02:18:13 -07:00
Chris Lu
0f10833e0b fix compilation 2020-03-24 00:16:12 -07:00
Chris Lu
3794f2721f fix logic 2020-03-24 00:09:12 -07:00
Chris Lu
443b229ef7 disable chunk file output 2020-03-24 00:08:02 -07:00
Chris Lu
bb9b97e2b4 add comments 2020-03-23 23:07:11 -07:00
Chris Lu
8047ec2f51 shell: fs.meta.save add option to export all fileIds for all files 2020-03-23 22:54:46 -07:00
Chris Lu
782d776d2a refactoring 2020-03-23 22:54:02 -07:00
Chris Lu
38e73463f1 fix du block and byte couting 2020-03-23 21:37:04 -07:00
Chris Lu
b51fa81f0e fix directory checking 2020-03-23 21:36:39 -07:00
Chris Lu
e666aeece2 simplify parsing filer host and port 2020-03-23 21:26:15 -07:00
Chris Lu
d151185b7e shell: desupport filer url in the arguments 2020-03-23 20:46:17 -07:00
Chris Lu
40601953bf comments 2020-03-23 02:54:49 -07:00
Chris Lu
daeb18be30 scaffold: fix master.toml typo 
fix https://github.com/chrislusf/seaweedfs/issues/1247
 2020-03-23 01:45:43 -07:00
Chris Lu
b97768c51c refactoring 2020-03-23 01:30:22 -07:00
Chris Lu
45ee3736aa refactoring 2020-03-23 01:25:38 -07:00
Chris Lu
cddcb5cafe same booleans as before 2020-03-23 01:25:23 -07:00
Chris Lu
f84c288852 purge 2020-03-23 01:14:55 -07:00
Chris Lu
a5aa8be19b Update webdav_server.go 2020-03-23 01:14:42 -07:00
Chris Lu
7f1e3c843d refactoring 2020-03-23 01:14:21 -07:00
Chris Lu
c07bcd5065 refactoring 2020-03-23 00:30:02 -07:00
Chris Lu
654a69ff52 refactoring 2020-03-23 00:06:24 -07:00
Chris Lu
c0f0fdb3ba refactoring 2020-03-23 00:01:34 -07:00
Chris Lu
fbca6b29bd refactoring 2020-03-22 23:52:55 -07:00
Chris Lu
e39e78ea8d remove println 2020-03-22 18:37:12 -07:00
Chris Lu
c16dc57a58 fix max volume count reporting 2020-03-22 18:33:19 -07:00
Chris Lu
d6412f27a0 logging 2020-03-22 18:32:56 -07:00
Chris Lu
35208711e5 logging 2020-03-22 18:32:49 -07:00
Chris Lu
e32999108a add auto configured volume max count help messge 2020-03-22 16:27:15 -07:00
Chris Lu
3137777d83 volume: automatically detect max volume count 2020-03-22 16:21:42 -07:00
Chris Lu
0bf148f49d logging 2020-03-22 16:19:00 -07:00
Chris Lu
bda13ed593 commented prints 2020-03-22 13:09:51 -07:00
Chris Lu
74e23e8d64 FUSE mount: rename correctly across folders 2020-03-22 13:09:33 -07:00
Chris Lu
64000f5c45 FUSE mount: rename under root 
fix https://github.com/chrislusf/seaweedfs/issues/1242
 2020-03-22 13:09:16 -07:00
Chris Lu
9f2c9b666b adjust log level 2020-03-22 01:39:08 -07:00
Chris Lu
ae2ee379c0 consistent 64bit size 2020-03-22 01:37:46 -07:00
Chris Lu
2bdd936fb6 purge code 2020-03-22 01:33:40 -07:00
Chris Lu
e93e986727 webdav: stream read 2020-03-22 01:30:27 -07:00
Chris Lu
65d2ea9fb0 FUSE mount: stream read data with buffer 
fix https://github.com/chrislusf/seaweedfs/issues/1244
 2020-03-22 01:00:36 -07:00
Chris Lu
82bfad5b86 refactoring 2020-03-21 23:48:11 -07:00
Chris Lu
6ff9e2835e filer: skip resizing image if width or height larger than original image 
fix https://github.com/chrislusf/seaweedfs/issues/1239#issuecomment-602140779
 2020-03-21 22:16:00 -07:00
Chris Lu
7c111f7b75 FUSE mount: make "nonempty" optional 
https://github.com/chrislusf/seaweedfs/issues/1094
 2020-03-21 19:14:25 -07:00
Chris Lu
3b3e063f25 1.67 2020-03-20 23:42:49 -07:00
Chris Lu
d3e4a31058 filer: fix where deletion can miss under high concurrency 2020-03-20 23:39:32 -07:00
Chris Lu
cbfe31a9a8 idx file sync before compaction 2020-03-20 23:38:46 -07:00
Chris Lu
3505b06023 report deletion error in the log 2020-03-20 23:38:24 -07:00
Chris Lu
4bc19f1e52 master: fix connections count reporting 2020-03-20 22:41:37 -07:00
Chris Lu
e1b5ac5c04 1.66 2020-03-20 20:39:48 -07:00
Chris Lu
308688c8d0 filer: add back image resizing capability 2020-03-20 20:31:11 -07:00
Chris Lu
f251d03673 refactoring 2020-03-20 15:46:16 -07:00
Chris Lu
11372dd548 refactoring 2020-03-20 15:38:04 -07:00
Chris Lu
c4bea45099 S3 API: fix DeleteMultipleObjectsHandler 
fix https://github.com/chrislusf/seaweedfs/issues/1241
 2020-03-20 14:17:31 -07:00
Chris Lu
165b0d22a4 1.65 2020-03-20 00:05:42 -07:00
Chris Lu
81797a059a volume: sync volume file right before compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1237
 2020-03-19 23:54:52 -07:00
Chris Lu
709f231e23 tiered storage: add s3 endpoint for private s3 implementation 
fix https://github.com/chrislusf/seaweedfs/issues/1238
 2020-03-19 21:13:56 -07:00
Chris Lu
d848d08944 use public url also for deletions 2020-03-18 11:16:45 -07:00
Chris Lu
6566c8e114 weed upload: add usePublicUrl option 
fix https://github.com/chrislusf/seaweedfs/issues/1236
 2020-03-18 10:50:53 -07:00
Chris Lu
db9854fa0a 1.64 2020-03-17 10:58:35 -07:00
Chris Lu
bec6ec7db6 go fmt 2020-03-17 10:01:55 -07:00
Chris Lu
b964bbab3d fix compilation 2020-03-17 10:01:24 -07:00
Chris Lu
c3cb6fa1d7 volume: compaction can cause readonly volumes 
address https://github.com/chrislusf/seaweedfs/issues/1233
 2020-03-17 09:43:57 -07:00
Chris Lu
f9b3750ad1 volume: handle repeated writes response 2020-03-15 04:16:00 -07:00
Chris Lu
22400c6633 consistent gzip logic 
local store gzip same as replicated writes
 2020-03-15 04:15:40 -07:00
Chris Lu
560df51def refactoring 2020-03-15 03:11:26 -07:00
Chris Lu
7edbee6f57 volume: proxy writes to remote volume server, with replication or not 
the panic is triggered by uploading a file to a volume server not holding the designated replica.
2020-03-15 10:20:14.365488 I | http: panic serving 127.0.0.1:57124: runtime error: invalid memory address or nil pointer dereference
goroutine 119 [running]:
net/http.(*conn).serve.func1(0xc0001a8000)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:1772 +0x139
panic(0x2316fe0, 0x3662900)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/runtime/panic.go:973 +0x396
github.com/chrislusf/seaweedfs/weed/topology.getWritableRemoteReplications(0xc00009c000, 0x2, 0x7ffeefbffbd2, 0xe, 0x0, 0xa, 0x0, 0x0, 0xbb4bf1f7)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/topology/store_replicate.go:157 +0x53
github.com/chrislusf/seaweedfs/weed/topology.ReplicatedWrite(0x7ffeefbffbd2, 0xe, 0xc00009c000, 0xc000000002, 0xc000472750, 0xc0001b2200, 0x0, 0x1, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/topology/store_replicate.go:29 +0xc7
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).PostHandler(0xc0001513f0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volume_server_handlers_write.go:52 +0x56f
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).privateStoreHandler(0xc0001513f0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volume_server_handlers.go:37 +0x21f
net/http.HandlerFunc.ServeHTTP(0xc0004420e0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2012 +0x44
net/http.(*ServeMux).ServeHTTP(0xc0001fc800, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2387 +0x1a5
net/http.serverHandler.ServeHTTP(0xc0001781c0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc0001a8000, 0x2934420, 0xc000212400)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2933 +0x35c
Eg:
server A (datacenter 1) and server B (datacenter 2) hold replica (100) for volume 1.
If you upload a file with a key 1,xxxxx to server C (datacenter 3) will trigger the panic on server C.
The server C should either proxy upload file to the correct volume server or should return an HTTP error code and not panic.
 2020-03-15 02:50:42 -07:00
Chris Lu
7b37178716 filer: close stores if interrupted 2020-03-14 20:30:26 -07:00
Chris Lu
d022b6bc0e fix compilation 2020-03-14 16:32:16 -07:00
Chris Lu
0576a27f44 protect against possible nil 
which is unlikely to happen though
 2020-03-14 16:06:06 -07:00
Chris Lu
cc52e8ca87 1.63 2020-03-14 14:25:07 -07:00
Chris Lu
ee635dcc36 refactoring 2020-03-14 14:08:00 -07:00
Chris Lu
18cda6adbb cleaner fix 2020-03-14 13:55:32 -07:00
Chris Lu
17efa361f9 fix possible nil 2020-03-14 13:53:03 -07:00
Chris Lu
91e0a987e4 1.62 2020-03-14 01:03:59 -07:00
Chris Lu
e2e691d9c2 clean up, add test 2020-03-13 23:53:15 -07:00
Chris Lu
3cc9e85895 volume: vacuum pass preallocate variable 2020-03-13 16:17:44 -07:00
Chris Lu
4ce23e8809 fix compilation error 2020-03-13 15:51:38 -07:00
Chris Lu
c90eb0da1f volume: handling readonly volumes after compaction 
ensure readonly volumes are not added as writable
 2020-03-13 15:41:27 -07:00
Chris Lu
e85da50ad4 remove tikv, move its support to "tikv" branch 2020-03-11 20:59:00 -07:00
Chris Lu
d439d83772 volume: follow compactionBytePerSecond 
related to https://github.com/chrislusf/seaweedfs/issues/1108
 2020-03-11 10:32:17 -07:00
Chris Lu
4237a813cc skip printouts 2020-03-11 01:13:54 -07:00
Chris Lu
ebc739afb5 filer store postgres: skip empty parameters 
better fix https://github.com/chrislusf/seaweedfs/issues/1227
 2020-03-11 01:13:54 -07:00
HongyanShen
81610ed006 fix: #1226 2020-03-11 14:37:14 +08:00
Chris Lu
60f5f05c78 1.61 2020-03-10 13:52:49 -07:00
Chris Lu
7f3ce7b6d6 adjust tikv build env 2020-03-10 13:36:45 -07:00
Chris Lu
d3e1ef64b3 ensure error checking on windows 2020-03-10 13:33:26 -07:00
Chris Lu
88a110e67e release file handle 2020-03-10 13:32:22 -07:00
Chris Lu
43e6261579 volume: fix memory leak 2020-03-10 13:31:23 -07:00
Chris Lu
d214cefc2e filer: list diretory pagination make up for the expired entries 2020-03-09 23:28:01 -07:00
Chris Lu
e6de42f888 1.60 2020-03-09 22:32:40 -07:00
Chris Lu
e73b0c2457 remove not ready status 2020-03-09 22:31:14 -07:00
Chris Lu
0871d2cff0 volume: fix memory leak during compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1222
 2020-03-09 22:29:02 -07:00
Chris Lu
14ae33d642 adjust logging 2020-03-09 10:36:03 -07:00
Chris Lu
89eb05b50f filer: support TTL for all filer stores 2020-03-09 01:02:01 -07:00
Chris Lu
8a899992f2 filer: fix ttl parsing 
fix https://github.com/chrislusf/seaweedfs/issues/1225
 2020-03-09 00:16:10 -07:00
Chris Lu
c32f95c380 1.59 2020-03-08 22:02:34 -07:00
Chris Lu
3934c69757 clean up 2020-03-08 21:54:47 -07:00
Chris Lu
2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu
5ac6297c68 adjust parameter names 2020-03-08 17:03:37 -07:00
Chris Lu
e04c1a1361 filer: remove chunk manifest file support 
This is not needed for filer
 2020-03-08 17:03:27 -07:00
Chris Lu
1dc30214cb mark encryptVolumeData as not ready 2020-03-08 17:02:47 -07:00
Chris Lu
9b3109a5d8 filer: processing all response headers, no pass through to volume server 
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
 2020-03-08 15:42:44 -07:00
Chris Lu
11fceaf2f7 refactoring 2020-03-07 18:09:30 -08:00
Chris Lu
1a75fbfce0 refactoring 2020-03-07 18:06:48 -08:00
Chris Lu
afb20de14c breaks dependency loop 2020-03-07 17:01:39 -08:00
Chris Lu
8645283a7b fuse mount: avoid lookup nil entry 
fix https://github.com/chrislusf/seaweedfs/issues/1221
 2020-03-07 16:51:46 -08:00
Chris Lu
936e7cdbfb pass in filer.copy cipher option 2020-03-07 12:46:00 -08:00
Chris Lu
9228ff192c encryption works 2020-03-07 11:08:57 -08:00
Chris Lu
0375ce2c2e filer: set mime type from volume server 2020-03-07 07:25:15 -08:00
Chris Lu
1ae83c2938 go fmt 2020-03-07 06:12:57 -08:00
Chris Lu
564629444b format 2020-03-07 06:12:20 -08:00
Chris Lu
8c3e25b38a mimee type default "application/octet-stream" to empty string 2020-03-07 06:08:08 -08:00
Chris Lu
dba35404e4 filer: HEAD response add content type 2020-03-07 06:08:08 -08:00
Chris Lu
ea1169dc80 filer cipher: single chunk http POST and PUT and read 2020-03-07 06:08:08 -08:00
Chris Lu
e3b8bf5588
Merge pull request #1219 from song-zhang/master 
schedule new volume by free volume number of nodes
 2020-03-07 05:54:06 -08:00
zhangsong
40f70481cd schedule new volume by free volume number of nodes 2020-03-07 21:41:42 +08:00
Chris Lu
13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu
31c481e3fc fix typo 2020-03-05 16:00:20 -08:00
Chris Lu
465d18930b filer redis: directory listing follow ttl 
fix https://github.com/chrislusf/seaweedfs/issues/1217
 2020-03-05 10:35:21 -08:00
Chris Lu
e031605248 purge code 2020-03-04 00:44:27 -08:00
Chris Lu
f90c43635d refactoring 2020-03-04 00:39:47 -08:00
Chris Lu
9b7dae8037 filer: fix uploaded chunked file size reporting bug. 2020-03-02 20:27:14 -08:00
Chris Lu
7335e62199 volume: PUT also conditionally gzip compress 2020-03-01 23:37:02 -08:00
Chris Lu
410bce3925 go fmt 2020-03-01 22:39:08 -08:00
Chris Lu
dc40413847 fix error 2020-03-01 22:21:13 -08:00
Chris Lu
6a8484b4ae master able to list all master clients by type 2020-03-01 22:13:47 -08:00
Chris Lu
0ca68a2a6d WIP 2020-03-01 22:13:15 -08:00
Chris Lu
4532640ffd 1.58 2020-03-01 17:59:12 -08:00
Chris Lu
151114ff59 volume: fix readonly status reporting 2020-03-01 17:57:37 -08:00
Chris Lu
1df13d0d6d adjust instructions 2020-02-27 13:15:21 -08:00
Chris Lu
f87f2045b3 delete old file first on windows 
fix https://github.com/chrislusf/seaweedfs/issues/1210
 2020-02-27 00:59:35 -08:00
Chris Lu
757c7d67ad avoid nil DataBackend 2020-02-27 00:07:25 -08:00
Chris Lu
555413d9fc weed queue starts 2020-02-27 00:07:13 -08:00
Chris Lu
430b5a49e2 unused code 2020-02-26 19:35:00 -08:00
Chris Lu
8c1da71402 remove dead code 2020-02-26 17:27:49 -08:00
Chris Lu
ed0acd1722 go fmt 2020-02-26 16:52:57 -08:00
Chris Lu
37a3628b2e grpc add PermitWithoutStream to client and server options 2020-02-26 16:51:38 -08:00
Chris Lu
556dd76303 s3: wait to connect to filer 2020-02-26 16:49:47 -08:00
Chris Lu
0156e2975a mount: add mode to run external to SeaweedFS container cluster 2020-02-26 16:46:01 -08:00
Chris Lu
543cf1c80e fix compilation error 2020-02-25 23:59:30 -08:00
Chris Lu
b1f377f822 mount: fail fast when connecting to filer 
fix https://github.com/chrislusf/seaweedfs/issues/1034
 2020-02-25 23:51:34 -08:00
Chris Lu
36d6595658 remove ctx 2020-02-25 22:38:27 -08:00
Chris Lu
fd9612d66e remove ctx 2020-02-25 22:37:54 -08:00
Chris Lu
86cce3eb58 fix test 2020-02-25 22:29:01 -08:00
Chris Lu
97ab8a1976 remove ctx if possible 2020-02-25 22:23:59 -08:00
Chris Lu
892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu
bd3254b53f adjust logging 2020-02-25 17:24:08 -08:00
Chris Lu
0841bedb15 move filer assign volume grpc errror to response 2020-02-25 17:15:09 -08:00
Chris Lu
bc38b72a20 s3: implemented DeleteMultipleObjects 2020-02-25 14:38:36 -08:00
Chris Lu
35dde56711 refactoring 2020-02-25 12:58:45 -08:00
Chris Lu
986d63cd5e s3: increase list objects or parts limit from 1000 to 10000 
10x of Amazon S3 limits
 2020-02-25 11:45:40 -08:00
Chris Lu
7d10fdf737 fix directory lookup nil 2020-02-25 11:13:06 -08:00
Chris Lu
e86da5a491 minor 2020-02-25 00:42:48 -08:00
Chris Lu
0644d63748 shell: add commands for bucket 2020-02-24 23:30:01 -08:00
Chris Lu
6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu
5bcb44eda9 filer: move buckets folder to filer.toml since it should not be changed often 2020-02-24 14:42:57 -08:00
Chris Lu
d8dec2323b s3: move buckets folder configuration to filer 2020-02-24 14:34:14 -08:00
Chris Lu
b06b7ca6e6 adjust UI 2020-02-23 22:01:03 -08:00
Yoni Nakache
0d60bb4427
display bug, header is Free but data was Used 2020-02-24 07:47:27 +02:00
Chris Lu
e96f85ed8a make pb files 2020-02-23 18:06:34 -08:00
Chris Lu
346a1cf0b9
Merge pull request #1207 from LazyDBA247-Anyvision/update-stats-and-status-volume-server 
Update stats and status volume server
 2020-02-23 18:05:25 -08:00
Chris Lu
bbb163c5ad add for https://github.com/chrislusf/seaweedfs/pull/1207 2020-02-23 18:04:31 -08:00
LazyDBA247-Anyvision
4ff513d64d staus route: add DiskStatuses for disk in the volume server status 
whem monitoring server, better to know the status of the disks & volumes in a single route.
 2020-02-23 23:27:09 +02:00
LazyDBA247-Anyvision
ab91dbe1d7 fillInDiskStatus: update function to calculate new stats 2020-02-23 23:25:56 +02:00
LazyDBA247-Anyvision
ec8619176a volume server protobuf: add precentFree & percentUsage to the DiskStatus 2020-02-23 23:24:24 +02:00
Yoni Nakache
cd4c9a365b
DiskStats: adding Total & Percent Usage 
making relevant data visible and readable.
 2020-02-23 22:33:47 +02:00
Chris Lu
6ea394b802 1.57 2020-02-22 21:34:49 -08:00
Chris Lu
4ed6b584e2 s3: access control limited by bucket 2020-02-22 21:34:18 -08:00
Chris Lu
e83bfd0a35 adjust log level 2020-02-22 21:23:30 -08:00
Chris Lu
48b7ad5fa8 s3: deny anonymous type 2020-02-22 14:01:04 -08:00
Chris Lu
f55f49970f move volume server stats to grpc 2020-02-21 21:45:03 -08:00
Chris Lu
3a1d017de2 shell: ensure dc and rack does not change for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1203
 2020-02-21 21:23:25 -08:00
Chris Lu
621cdbdf58 filer: avoid possible timeouts for updates and deletions 2020-02-20 15:44:17 -08:00
Chris Lu
45156cc2fe add test cases for replication=011 2020-02-20 01:21:11 -08:00
Chris Lu
b07089cff2 webdav out of "unstable" 2020-02-18 19:38:59 -08:00
Chris Lu
dbabdd418e add unhandled request auth type 
fix

2020-02-18 11:43:57.396699 I | http: panic serving 172.28.0.43:50658: runtime error: invalid memory address or nil pointer dereference
goroutine 595 [running]:
net/http.(*conn).serve.func1(0xc0001fe3c0)
    /usr/lib/go/src/net/http/server.go:1767 +0x13b
panic(0x55c4e35f3820, 0x55c4e48b3c40)
    /usr/lib/go/src/runtime/panic.go:679 +0x1b6
github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).authRequest(0xc0004b84e0, 0xc000115900, 0xc0000bb650, 0x1, 0x1, 0x55c4e399d740)
    /go/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:143 +0x11c
github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).Auth.func1(0x55c4e3994c40, 0xc0007808c0, 0xc000115900)
    /go/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:111 +0x5e
net/http.HandlerFunc.ServeHTTP(0xc0004b87e0, 0x55c4e3994c40, 0xc0007808c0, 0xc000115900)
    /usr/lib/go/src/net/http/server.go:2007 +0x46
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0004ba000, 0x55c4e3994c40, 0xc0007808c0, 0xc000115700)
    /root/go/pkg/mod/github.com/gorilla/mux@v1.7.3/mux.go:212 +0xe4
net/http.serverHandler.ServeHTTP(0xc00011e0e0, 0x55c4e3994c40, 0xc0007808c0, 0xc000115700)
    /usr/lib/go/src/net/http/server.go:2802 +0xa6
net/http.(*conn).serve(0xc0001fe3c0, 0x55c4e399d680, 0xc000894180)
    /usr/lib/go/src/net/http/server.go:1890 +0x877
created by net/http.(*Server).Serve
    /usr/lib/go/src/net/http/server.go:2927 +0x390
 2020-02-18 09:16:04 -08:00
Chris Lu
443a68484a 1.56 2020-02-17 12:54:45 -08:00
Chris Lu
0a57db7026
Merge pull request #1200 from chrislusf/add_s3_acl 
Add s3 ACL
 2020-02-17 12:53:30 -08:00
Chris Lu
4cdde5f569 configuration stores the identity list 2020-02-17 12:31:59 -08:00
Chris Lu
9232d3ac68 ignore error when counting closed connections 2020-02-14 10:28:02 -08:00
Chris Lu
96c1ae8471 refactoring the close http response 2020-02-14 09:46:36 -08:00
Chris Lu
cf5064d702 properly close http response 2020-02-14 09:09:15 -08:00
Chris Lu
9b6296e77a volume: add grpc file read operation 
This is added more for performance benchmarking.
 2020-02-14 00:37:32 -08:00
panyc16
7b3764fd9e make loadExistingVolume() return bool 2020-02-13 15:19:00 +08:00
Chris Lu
62c34454d8 1.55 2020-02-11 23:00:31 -08:00
Chris Lu
83836f4299 filer: recursively batch delete file chunks 
fix https://github.com/chrislusf/seaweedfs/issues/1197
 2020-02-11 22:54:10 -08:00
Chris Lu
2a6db0fd43 volume: return 204 for unchanged file uploads 
fix https://github.com/chrislusf/seaweedfs/issues/1196
 2020-02-11 09:45:02 -08:00
Chris Lu
33b3bd467c Revert "HEAD operation changes to fasthttp" 
This reverts commit 58f126fd27.
 2020-02-10 13:43:53 -08:00
Chris Lu
58f126fd27 HEAD operation changes to fasthttp 2020-02-10 09:13:29 -08:00
Chris Lu
29945fad51 1.54 2020-02-09 18:07:17 -08:00
Chris Lu
b4abe3c081 unused 2020-02-09 18:02:17 -08:00
Chris Lu
f3ce3166ad add streaming v4 2020-02-09 17:42:17 -08:00
Chris Lu
b90ad6f452 add v2 support 2020-02-09 16:02:05 -08:00
Chris Lu
e7b63d50b1 add s3.config to server.go 2020-02-09 14:31:51 -08:00
Chris Lu
9ed364f053 support acl 2020-02-09 14:30:02 -08:00
Chris Lu
8d94564f41 refactor 2020-02-04 21:16:34 -08:00
Chris Lu
c1288e9eb4 volume: sdx generation uses memdb instead of compactMap 
fix https://github.com/chrislusf/seaweedfs/issues/1194
 2020-02-04 21:12:09 -08:00
Chris Lu
08e4702542 sync before closing index file 
merge from 0181f87d91
 2020-02-04 10:38:12 -08:00
Chris Lu
3a35632d58 sync before closing file 
merge changes from 1747fc2d52
 2020-02-04 10:37:14 -08:00
Chris Lu
dc786a63ac master: add configurable volume growth toml setting 2020-02-03 18:15:16 -08:00
Chris Lu
382ff218d3 filer: set file size, streaming chunk file uploading 
fix https://github.com/chrislusf/seaweedfs/issues/1193
 2020-02-03 17:04:06 -08:00
Chris Lu
40ae533fa3 shell: add volume.configure.replication to change replication for a volume 
fix https://github.com/chrislusf/seaweedfs/issues/1192
 2020-02-02 15:37:23 -08:00
Chris Lu
fb19263a71 fix build 2020-01-31 00:59:48 -08:00
Chris Lu
beb0a1599e 1.53 2020-01-31 00:32:09 -08:00
Chris Lu
b2743afaee remove println 2020-01-31 00:21:18 -08:00
Chris Lu
a80ecbfe84 s3: add s3 copy 
fix https://github.com/chrislusf/seaweedfs/issues/1190
 2020-01-31 00:11:12 -08:00
Chris Lu
3b043ead49
Merge pull request #1189 from eryx67/compression-wav 
Support wav files compression
 2020-01-29 22:37:35 -08:00
Chris Lu
abf90ad7b7 remove testing for viper 2020-01-29 21:38:53 -08:00
Chris Lu
20b1fb9ab7 fix test 2020-01-29 21:24:06 -08:00
eryx67
2a87087590 support wav files compression 2020-01-30 10:11:47 +05:00
Chris Lu
8925f3305d adjust example 2020-01-29 09:11:07 -08:00
Chris Lu
d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
Chris Lu
27b94cb65b fix wrong url 
fix https://github.com/chrislusf/seaweedfs/issues/1187
 2020-01-28 00:49:47 -08:00
Chris Lu
f724057051 reduce logs 2020-01-27 21:20:41 -08:00
Chris Lu
c8d543ef51 avoid retry delay 2020-01-27 00:58:26 -08:00
Chris Lu
081bc1ea25 mount: able to write to any part of a file 2020-01-27 00:54:52 -08:00
Chris Lu
2f6bb57979 logging 2020-01-27 00:54:21 -08:00
Chris Lu
234f69452b 1.52 2020-01-26 21:04:40 -08:00
Chris Lu
72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu
08e4b56a8a mount: able to handle large git clone 2020-01-26 13:01:11 -08:00
Chris Lu
19a05ad174 add test cases 2020-01-26 12:47:23 -08:00
Chris Lu
835da19c09 add logging 2020-01-26 00:50:18 -08:00
Chris Lu
c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu
ccf3859152 log 2020-01-25 08:16:43 -08:00
Chris Lu
4433c7bdca logs 2020-01-25 07:34:09 -08:00
Chris Lu
8cccccce9f mount: reuse the entry object for the new directory 
avoid mkdir and then query for the same directory
reduces these "context canceled" issues

attr read directory:"/seaweedfs/other/java/s3copier/src/main/java/com" name:"seaweedfs" : rpc error: code = Canceled desc = context canceled
 2020-01-25 01:15:54 -08:00
Chris Lu
90e30c0170 log 2020-01-25 00:32:18 -08:00
Chris Lu
8a42aa8221 adjust logging 2020-01-25 00:31:53 -08:00
Chris Lu
28d1b34856 logging 2020-01-24 22:13:06 -08:00
Chris Lu
6e25acc681 add logging 2020-01-24 22:01:51 -08:00
Chris Lu
b6e6ca8595 adjust logs 2020-01-24 20:06:58 -08:00
Chris Lu
d4cde5df49 return empty response if not found 2020-01-24 18:07:34 -08:00
Chris Lu
3cdb27fafc adjust logging 2020-01-24 17:55:39 -08:00
Chris Lu
6107509c64 adjust logging 2020-01-24 17:26:18 -08:00
Chris Lu
e6f9f5da3a adjust logging 2020-01-24 17:26:03 -08:00
Chris Lu
24f62ebe76 minor changes 2020-01-24 10:02:53 -08:00
Chris Lu
5f1109f143 reduce log 2020-01-24 01:41:51 -08:00
Chris Lu
9d93c83654 mount: return IO error for mkdir 2020-01-24 01:41:31 -08:00
Chris Lu
107e8a56ea retry context canceled request 2020-01-24 01:40:51 -08:00
Chris Lu
2f75264ec7 mount: avoid use uint64 for greater than zero test 2020-01-23 22:12:57 -08:00
Chris Lu
c936a12afa mount: update the file size so file Attr() can read 
if file Attr() during file writes, the OS, at least for Mac, may try to resend the data to write, which usually are empty.
 2020-01-23 21:59:58 -08:00
Chris Lu
c2e589f202 mount: better combines connected intervals to write to volume servers 2020-01-22 23:00:04 -08:00
Chris Lu
6a5c037099 fix http range requests 2020-01-22 22:59:12 -08:00
Chris Lu
72e1aff237 add lock on dirty pages 2020-01-22 15:38:25 -08:00
Chris Lu
66ded8804a adjust logging 2020-01-22 15:37:59 -08:00
Chris Lu
6b48d246a5 mount: read data that is just written 
able read on data not flushed
multiple file open shares the same file handle

fix https://github.com/chrislusf/seaweedfs/issues/1182 on linux
 2020-01-22 13:42:03 -08:00
Chris Lu
09f4ceef3a mount: reuse previous file node 2020-01-22 11:43:43 -08:00
Chris Lu
d4e75a0d18 filer: option to create file only if it is new, O_EXCL 2020-01-22 11:42:40 -08:00
Chris Lu
9b01a99d9a adjust logging 2020-01-21 22:45:50 -08:00
Chris Lu
c8b2dac6c1 volume: avoid sharing volume dat file handle 
possibly help on https://github.com/chrislusf/seaweedfs/issues/1184
 2020-01-21 21:18:01 -08:00
Chris Lu
bb1be61602 1.51 2020-01-20 20:30:03 -08:00
Chris Lu
fca6152c3a fix print format 2020-01-20 20:24:23 -08:00
Chris Lu
a990ef2106 mount: fix problems found in issue 1182 
fix https://github.com/chrislusf/seaweedfs/issues/1182

always use the non-duplicated fs.Node
Forget() the fs.Node
Rename will also use the right fs.Node
Avoid using the same file handle for the same file
 2020-01-20 20:21:01 -08:00
Chris Lu
630f72f8c5 mount: able to trim a file 2020-01-20 00:00:08 -08:00
Chris Lu
1b0bfbaf59 refactoring 2020-01-19 23:59:46 -08:00
Chris Lu
2f15e93466 mount: rename also clear the cache 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-01-19 12:31:56 -08:00
Chris Lu
04019aa161 mount: deletion ignore old vid 2020-01-19 12:07:26 -08:00
Chris Lu
6a40c688e0 mount: modify file size 2020-01-19 12:07:04 -08:00
Chris Lu
39edcfde4c filer: set inode 
avoid "getcwd: No such file or directory"
 2020-01-19 12:06:19 -08:00
Chris Lu
e9bb0c179f adjust logs 2020-01-18 13:25:17 -08:00
Chris Lu
061a61e018 logs 2020-01-15 19:09:00 -08:00
Chris Lu
230a0e4a33 filer: no entry if not found 2020-01-15 19:08:54 -08:00
Chris Lu
e4b660699f correct comments 
fix https://github.com/chrislusf/seaweedfs/issues/1181
 2020-01-12 20:31:33 -08:00
Chris Lu
d1ab16b6e3 treat it as a single node cluster if empty raft server name 
possible fix for https://github.com/chrislusf/seaweedfs/issues/1118
 2020-01-10 00:37:44 -08:00
Chris Lu
30c7148020 1.50 2020-01-08 22:08:35 -08:00
Chris Lu
8dd260623e protect against nil deletion results 
fix https://github.com/chrislusf/seaweedfs/issues/1180
 2020-01-08 21:50:59 -08:00
Chris Lu
89e16bd2e8 skip error when draining reader 
fix https://github.com/chrislusf/seaweedfs/issues/1179
 2020-01-08 18:07:07 -08:00
Chris Lu
66ab09aa4a 1.49 2020-01-08 09:51:35 -08:00
Chris Lu
f8a20ef35e add the old way to compact as a comment 2020-01-08 09:45:42 -08:00
Chris Lu
288baf37fd saving .vif files correctly 2020-01-08 09:45:26 -08:00
Chris Lu
acf7ca7b93 volume: fix compaction 2020-01-08 09:45:03 -08:00
Chris Lu
943f4986ef fix possible nil 2020-01-08 08:49:18 -08:00
Chris Lu
9995d3bcb5 remove println 2020-01-07 21:49:28 -08:00
Chris Lu
dc9457fb87 fix compilation error 2020-01-07 09:03:47 -08:00
Chris Lu
aa73364a24 weed fix: add back .idx file generation 2020-01-06 16:29:59 -08:00
Chris Lu
4e731f1c8b volume: copy volumes also include .vif file 2020-01-04 11:28:29 -08:00
Chris Lu
df636e4c14 remove .vif files 2020-01-03 12:46:42 -08:00
Chris Lu
3eafec4b29 volume: add option to limit file size 2020-01-03 00:37:26 -08:00
Chris Lu
9c139e0601 update help message 2020-01-01 12:38:29 -08:00
Chris Lu
88a80ab557 fix help message 2020-01-01 12:37:38 -08:00
Chris Lu
aa1807e082 1.48 2020-01-01 09:27:02 -08:00
Chris Lu
86d4b18eb5 filer: optional recursive deletion 
fix https://github.com/chrislusf/seaweedfs/issues/1176
 2019-12-31 11:52:54 -08:00
Chris Lu
09043c8e5a fs: synchronized meta file writing 
fix https://github.com/chrislusf/seaweedfs/issues/1175
 2019-12-29 20:19:51 -08:00
Chris Lu
509f314350 keep .vif file for ec to normal volume conversion 2019-12-28 21:52:55 -08:00
Chris Lu
672868b460 always create .vif file 2019-12-28 21:52:06 -08:00
Chris Lu
03ab0b1b9a correct .vif file creation 2019-12-28 21:40:12 -08:00
Chris Lu
b6f7033478 go fmt 2019-12-28 21:37:29 -08:00
Chris Lu
9134092a8d endure creating .vif files 2019-12-28 21:36:15 -08:00
Chris Lu
0cc8347a3a go fmt 2019-12-28 21:13:10 -08:00
Chris Lu
8fbc0a9163 fix edge cases 2019-12-28 12:59:31 -08:00
Chris Lu
37b64a50b4 ec: generate and copy .vif file 2019-12-28 12:44:59 -08:00
Chris Lu
c06f7eb48a load volume info from .vif file, use superblock as a backup 2019-12-28 12:28:58 -08:00
Chris Lu
f4a74e03d1 refactoring: separating .vif from tier file loading 2019-12-28 11:35:27 -08:00
Chris Lu
2000284435 rename volume tier info to volume info 2019-12-28 11:21:49 -08:00
Chris Lu
e8b357fd53 rename from .tier to .vif 2019-12-28 11:17:39 -08:00
Chris Lu
b3c83f9404 fix spelling 2019-12-28 11:16:10 -08:00
Chris Lu
1fd8926ac7 ignore draining error 2019-12-25 21:39:33 -08:00
Chris Lu
48d28d3eb2 tier: support remote file attributes and remember the file extension 2019-12-25 21:37:24 -08:00
divinerapier
4cbb6fa199
feat: drains http body if buffer is too small 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-12-26 11:28:43 +08:00
divinerapier
84640d07b7
fix: handle errors for ReadUrl 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-12-26 10:07:07 +08:00
Chris Lu
1346437d71
Merge pull request #1171 from stlpmo-jn/redundant_type_conversion 
remove the redundant type conversion
 2019-12-25 17:02:05 -08:00
Chris Lu
35393b4a02 cleanly destroy remote files 2019-12-25 16:17:58 -08:00
Chris Lu
eecad6062a go fmt 2019-12-25 09:56:42 -08:00
Chris Lu
d960b3474a tier storage: support downloading the remote dat files 2019-12-25 09:53:13 -08:00
stlpmo
f3de4b6c18 remove the redundant type conversion 2019-12-25 10:13:45 +08:00
Chris Lu
3ebeae0c0b ec encode distribute ec data and parity shards evenly 2019-12-24 16:52:21 -08:00
Chris Lu
9ff72f616a go fmt 2019-12-24 14:56:16 -08:00
Chris Lu
efd2f50ede compaction changed to .idx based deletion 2019-12-24 14:55:50 -08:00
Chris Lu
7ba6be2312 volume ui add readonly status 2019-12-24 14:55:26 -08:00
Chris Lu
d8b39fe92a testing 2019-12-24 11:29:26 -08:00
Chris Lu
f42b5bd0f5 rename .sdb to .sdx 2019-12-24 10:19:12 -08:00
Chris Lu
abffe857a1 change btree map to in memory level db 2019-12-24 10:18:56 -08:00
Chris Lu
72a561ab7c refactoring 2019-12-24 08:13:01 -08:00
Chris Lu
9114f7f432
Merge pull request #1169 from stlpmo-jn/http_304_has_no_resp_body 
remove the http 304 response body
 2019-12-24 07:08:46 -08:00
Chris Lu
409a3fe41f
Merge pull request #1168 from stlpmo-jn/optimize_DeleteCollectionFromDiskLocation 
decouple the volume.Destroy() from the operation of unmountVolume()
 2019-12-24 07:08:04 -08:00
Chris Lu
37a836c8a7
Merge pull request #1167 from stlpmo-jn/volume_server_core_dump 
resolve issue : create volume on no free space disk, the volume server core dump
 2019-12-24 07:07:41 -08:00
stlpmo
f87ccbddb4 remove the http 304 response body 2019-12-24 17:56:50 +08:00
stlpmo
38e4b79125 decouple the volume.Destroy() from the operation of unmountVolume() 2019-12-24 17:20:34 +08:00
stlpmo
f300c35266 resolve issue : when the host disk full, the volume server will core dump 2019-12-24 16:36:15 +08:00
Chris Lu
1ad34a2487 ed.decode prefers servers with most data shards 2019-12-24 00:00:45 -08:00
Chris Lu
a18f62bbe7 only copy required shards 2019-12-23 18:06:13 -08:00
Chris Lu
c7d5a0b00c adjust help message 2019-12-23 17:58:47 -08:00
Chris Lu
8a2d529759 go fmt 2019-12-23 12:48:53 -08:00
Chris Lu
09ca936c78 shell: add ec.decode command 2019-12-23 12:48:20 -08:00
Chris Lu
dda5c6d3cb fmt 2019-12-22 04:31:36 -08:00
Chris Lu
a9b1ee18af default redis reads options 
details in https://github.com/chrislusf/seaweedfs/pull/1165
 2019-12-21 09:30:51 -08:00
Chris Lu
fcbd04c425 default to false in scaffold.go 2019-12-21 08:31:27 -08:00
Chris Lu
62b1a346ef set default 2019-12-20 20:57:38 -08:00
Chris Lu
cd47112c40
Merge pull request #1165 from dqsully/master 
Add readOnly and routeByLatency options to Redis Cluster Store
 2019-12-20 20:56:34 -08:00
Chris Lu
573b74f24f add SetDefault function for Configuration 2019-12-20 20:56:14 -08:00
Dakota Sullivan
f934a85d58
Add readOnly and routeByLatency options to scaffold.go 2019-12-20 19:27:15 -07:00
Dakota Sullivan
a0c9d9c0e7
Fix config property names 2019-12-20 19:23:01 -07:00
Chris Lu
71d55ec5cb adjust help message 2019-12-20 08:22:32 -08:00
Chris Lu
d0aa0c5086 shell: add fs.meta.cat to see file meta data content 
related to https://github.com/chrislusf/seaweedfs/issues/1163
 2019-12-20 08:22:18 -08:00
Chris Lu
ab966d7192 refactoring variable name 2019-12-19 00:44:46 -08:00
Chris Lu
f61de28c69 volume: add deletion capability for previously readonly volumes 2019-12-19 00:42:46 -08:00
Chris Lu
deb03ecfa3 remove duplicated checking 
already checked in store.go
 2019-12-18 23:59:53 -08:00
Chris Lu
36ddca9d1f filer: return not found 404 status if not found 
fix https://github.com/chrislusf/seaweedfs/issues/1160
 2019-12-18 21:04:40 -08:00
Chris Lu
ef3ae3cd41 clean up sorted index file 2019-12-18 20:59:15 -08:00
Dakota Sullivan
301bc2f69d
Add readOnly and routeByLatency options to Redis Cluster Store 2019-12-18 08:36:39 -07:00
Chris Lu
39810a9951 tiered volume is read only 2019-12-18 01:26:22 -08:00
Chris Lu
58f88e530c volume: use sorted index map for readonly volumes 2019-12-18 01:21:21 -08:00
Chris Lu
2f21beaccd loading xattr value 2019-12-17 21:45:32 -08:00
Chris Lu
014487b1a6 passing entry.Extended 2019-12-17 21:39:48 -08:00
Chris Lu
e707211dad passing entry.exended 2019-12-17 21:22:16 -08:00
Chris Lu
29eb333f8e fix comparing 2019-12-17 21:15:28 -08:00
Chris Lu
14242e6c70 passing xattr 2019-12-17 21:10:26 -08:00
Chris Lu
c42b023960 mount: invalidate xattr 2019-12-17 20:38:56 -08:00
Chris Lu
54369eccea 1.47 2019-12-17 15:11:01 -08:00
Chris Lu
6808bfec49 1.47 2019-12-17 15:08:45 -08:00
Chris Lu
9c781da34a mount: fix nil directory problem 2019-12-17 15:08:38 -08:00
Chris Lu
ee6952fddd twaking 2019-12-17 14:03:57 -08:00
Chris Lu
f27a1abf1c mount: fix dir nil 2019-12-17 13:50:32 -08:00
bs106
ce79a31448 Fix missing leading dot in extension 2019-12-17 09:47:12 +00:00
Chris Lu
5961e0d9eb fix logging 2019-12-15 22:24:06 -08:00
Chris Lu
33b4b1868b mount: support extended attributes 2019-12-15 21:07:01 -08:00
Chris Lu
ad9d5ff5d0 adjust UI 2019-12-15 20:57:17 -08:00
Chris Lu
392678f8f3 upload skipping mimetype if not needed 2019-12-15 20:57:08 -08:00
Chris Lu
e6ee421d61 fix compilation 2019-12-14 12:04:25 -08:00
Chris Lu
a631b2a9e9 1.46 2019-12-14 11:28:48 -08:00
Chris Lu
33da30d2b1 tweak 2019-12-14 11:04:20 -08:00
Chris Lu
de3eb7c380 update cache if updating the attributes 2019-12-13 11:14:09 -08:00
Chris Lu
357ea4b944 rm will clean the cache first 2019-12-13 11:08:26 -08:00
Chris Lu
deb249736e fix nill 2019-12-13 10:35:23 -08:00
Chris Lu
f59119cd1d mount: rename to dirListCacheLimit to configure cache for long format directory list 2019-12-13 10:22:24 -08:00
Chris Lu
3e0cd122d8 mount: directory listing cache ttl set to 5 minutes 2019-12-13 10:08:27 -08:00
Chris Lu
05c3b795dc filer: configurable directory list cache size 2019-12-13 10:05:43 -08:00
Chris Lu
bbb6ebc3c0 filer: DeleteFolderChildren for deleting large folders 2019-12-13 00:23:05 -08:00
Chris Lu
0fa1269bc7 filer: streaming file listing 2019-12-13 00:22:37 -08:00
Chris Lu
d0b423bbc0 filer: increase directory listing pagination size 2019-12-12 09:11:10 -08:00
Chris Lu
bf4b13612d adjust command name 2019-12-11 23:13:04 -08:00
Chris Lu
1f400489c6 filer: UI ensure a correct foler link 2019-12-11 23:09:01 -08:00
Chris Lu
4cba32d1d0 go fmt 2019-12-11 22:22:09 -08:00
Chris Lu
f51a6d8ba5 increase filer pagination limit 2019-12-11 22:21:37 -08:00
Chris Lu
c602073395 adjust logs 2019-12-11 21:40:33 -08:00
Chris Lu
1ddfbff2d3 deletion add option skipChunkDeletion 2019-12-11 14:58:22 -08:00
Chris Lu
f06b65a6fd filer: EXPERT add shouldDeleteChunks parameter for fast deletion 2019-12-11 07:50:53 -08:00
Yoni Nakache
9e45702334
Filer: Enable delete entries without flag volume 
same as happens on the s3 API on delete bucket,
to reduce the IO requests on the system, you can just clear recursively folders without flagging them in volume server (they will be removed on delete bucket/drop collection)
this will remove the user ability to retrieve data under these folders
 2019-12-11 11:30:19 +02:00
Chris Lu
40ff4eea14 fix nil problem 2019-12-10 23:13:14 -08:00
Chris Lu
356bd1b629 adjust text 2019-12-10 22:09:29 -08:00
Chris Lu
5847c5afff return meaningful error log 2019-12-10 22:09:29 -08:00
Chris Lu
3727d2488f
Merge pull request #1149 from song-zhang/improve-replications-consistency 
improve data consistency when replication write
 2019-12-09 20:13:03 -08:00
Chris Lu
10bd3c6b4b refactoring 2019-12-08 19:44:16 -08:00
Chris Lu
2b8e20f122 remove unused functions 2019-12-08 18:33:25 -08:00
zhangsong
5eb86aad47 improve data consistency when replication write 2019-12-08 22:21:35 +08:00
Chris Lu
e2a8cdfc48 webdav: fix writing file 
fix https://github.com/chrislusf/seaweedfs/issues/1114
 2019-12-07 07:56:51 -08:00
Chris Lu
0eda75fa2c add logging 2019-12-07 07:56:05 -08:00
Chris Lu
f81d43442b filer: speed up filer.meta.save by parallelizing 2019-12-06 22:20:59 -08:00
Chris Lu
f38f90b7ea cleanly remove .tier volume file 2019-12-06 22:19:47 -08:00
Chris Lu
6827cabfbb volume: dataFileAccessLock change to RW Lock 2019-12-06 06:59:57 -08:00
Chris Lu
81f424a630 go fmt 2019-12-06 06:57:32 -08:00
Chris Lu
916423e96a master: submit support "dataCenter" parameter 2019-12-05 15:51:24 -08:00
Chris Lu
e426bd541e grow volumes on volume servers with slots freed by cloud storage 2019-12-03 21:36:42 -08:00
Chris Lu
7ae8b1cc86 show volume info in volume.list in weed shell 2019-12-02 23:38:56 -08:00
Chris Lu
1becbce657 display remote volumes on volume server ui page 2019-12-02 23:23:54 -08:00
Chris Lu
70648d35ad go fmt 2019-12-02 20:49:58 -08:00
Chris Lu
6383b45bd0 add lock variable 2019-12-02 20:49:50 -08:00
Chris Lu
caae543a9f fix test 2019-12-02 15:54:24 -08:00
Chris Lu
126fb492d2 fix test 2019-12-02 15:52:33 -08:00
Chris Lu
ec8de250e2 tiered storage: can copy to s3, read from s3 
master not aware tiered volume yet, file assigning is not working yet
 2019-12-02 15:08:31 -08:00
Chris Lu
228231f3d7 minor 2019-12-02 15:08:31 -08:00
zhangsong
e83c36e26f fix the bug of volume never be vacuumed 2019-12-02 13:25:32 +08:00
Chris Lu
0da7b894cc pass backend config from master to volume servers 2019-11-29 01:05:09 -08:00
Chris Lu
a9dbd52e06
Merge pull request #1145 from stlpmo-jn/volume_graceful_stop 
let volume server graceful stop
 2019-11-29 00:11:57 -08:00
stlpmo
ea9d1ebd2f resolve the Duration.Milliseconds() is not exist in golang v11 & v12 2019-11-29 15:17:17 +08:00
Chris Lu
ea9a7e61d0 only send metrics address during on the first response 2019-11-28 20:48:08 -08:00
Chris Lu
61bc1d6ffd tweaking 2019-11-28 18:47:51 -08:00
Chris Lu
09c6d8fa32 fix test 2019-11-28 18:44:27 -08:00
Chris Lu
f60154f330 master load backend storage config from master.toml 2019-11-28 18:33:18 -08:00
Chris Lu
641b92f53c fix scaffold error 2019-11-28 18:04:25 -08:00
Chris Lu
ee2072dff9 volume: avoid inifinite loop reading ec volume info if error happens 2019-11-27 23:07:53 -08:00
Chris Lu
0b822a6fb5
Merge pull request #1142 from iliul/wip-fuse-mount 
fuse: check mount point available before do runmount
 2019-11-27 19:22:51 -08:00
Lei Liu
13b6f09fea fuse: adjust linux specific logic 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-11-28 10:21:31 +08:00
Chris Lu
586798ecc0 go fmt 2019-11-27 12:34:57 -08:00
Chris Lu
46294b42e6 filer: support memsql 
fix https://github.com/chrislusf/seaweedfs/issues/1141
 2019-11-27 12:34:03 -08:00
Lei Liu
1ba3456ac2
go fmt 
Signed-off-by: Lei Liu <liul.stone@gmail.com>
 2019-11-27 23:08:51 +08:00
Lei Liu
4c87b222f1
fuse: check mount point available before do runmount 
1. Use more readable mount point information
2. Fix some typos

eg:

$ df -Th
Filesystem              Type            Size  Used Avail Use% Mounted on
localhost:8888:/        fuse.seaweedfs  206G   512  206G   1% /mnt/weedfs

$ mount | grep weedfs
localhost:8888:/ on /mnt/weedfs type fuse.seaweedfs (rw,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)

Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-11-27 22:56:44 +08:00
Chris Lu
0f9ba84274 s3 2019-11-27 03:09:45 -08:00
Chris Lu
67e5f5b558 typo 2019-11-27 03:09:45 -08:00
Lei Liu
2700ac6d1f filer: Add attr.mode in query string 
Signed-off-by: Lei Liu <liul.stone@gmail.com>
 2019-11-27 13:57:29 +08:00
Chris Lu
5f6b360eb7 adjust error message 2019-11-24 23:13:40 -08:00
Chris Lu
2cbe79acda fix typo in command help 2019-11-24 22:17:43 -08:00
Chris Lu
0fca1ae776 adjust build 2019-11-22 23:19:49 -08:00
Chris Lu
bc6cd93431 skip build for 32 bit computers 2019-11-22 22:52:21 -08:00
Chris Lu
249b9d8d5d 1.45 2019-11-22 14:51:16 -08:00
Chris Lu
e4c63ff2f8 go fmt 2019-11-22 14:51:09 -08:00
Chris Lu
fb3c9ed80d shell: fs.meta.save missed directory data with more than 1024 entries 
fix https://github.com/chrislusf/seaweedfs/issues/1130
 2019-11-22 01:39:50 -08:00
divinerapier
5656d43264
can not break out of for-select block 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-11-20 08:25:29 +08:00
Chris Lu
9711a6ffaa WIP 2019-11-18 19:24:37 -08:00
Chris Lu
ba73c053c3 change uuid package 
fix https://github.com/chrislusf/seaweedfs/issues/1126
 2019-11-18 19:24:37 -08:00
chenwanli
295f3beed5 rm ldb and bdb directory 2019-11-19 09:35:06 +08:00
stlpmo
753138a3f7 test passed 2019-11-17 11:40:36 +08:00
Chris Lu
de0c983808 refactoring scaffolding for master.toml 2019-11-14 22:20:38 -08:00
Chris Lu
c6e8225a20 adjust memory mapped file size 
related to https://github.com/chrislusf/seaweedfs/pull/1087
 2019-11-12 21:05:48 -08:00
Chris Lu
66acc2c84f passing in averageShardsPerEcNode instead of expectedTotalEcShards 
addressing https://github.com/chrislusf/seaweedfs/issues/1111
 2019-11-12 15:36:18 -08:00
Chris Lu
40514c5362 add sortEcNodesByFreeslotsDecending and sortEcNodesByFreeslotsAscending 
addressing https://github.com/chrislusf/seaweedfs/issues/1111
 2019-11-12 13:47:36 -08:00
Chris Lu
717ec47fa2 avoid double counting ec slot 
addressing issue found in https://github.com/chrislusf/seaweedfs/issues/1111

The ec slots are already counted on the server side.
 2019-11-12 09:33:51 -08:00
Lei Liu
46755ea1e1 fix master maintenance logic 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-11-12 14:46:10 +08:00
Chris Lu
ee90236a97 Merge branch 'master' into refactoring_dat_backend 2019-11-11 21:30:14 -08:00
Chris Lu
d7852ebb3b
Merge pull request #1102 from stlpmo-jn/add_ETCD_sequencer 
Add etcd sequencer
 2019-11-11 12:43:24 -08:00
stlpmo
d07701fa75 delete unused function 2019-11-11 18:08:48 +08:00
stlpmo
62d393d6c9 ut pass 2019-11-11 10:52:21 +08:00
stlpmo
802a0eb3fe move from cmd-line to scaffold 2019-11-11 09:15:17 +08:00
Chris Lu
9c2f3b1b0f fix rack_not_found case 
fix rack_not_found in https://github.com/chrislusf/seaweedfs/issues/1111
 2019-11-10 12:06:41 -08:00
Chris Lu
e6ab75972f use constant 2019-11-10 12:05:58 -08:00
Chris Lu
471902cd51 change parameter name 2019-11-10 12:05:45 -08:00
zhangsong
61fa485700 add volume number param in assign operation 2019-11-10 20:11:03 +08:00
Chris Lu
85f8649320 refactor memory mapped file into backend storage 2019-11-09 00:10:59 -08:00
Chris Lu
c5c1d83d91 Merge branch 'master' into refactoring_dat_backend 2019-11-08 22:54:41 -08:00
Chris Lu
c34ffed43f go fmt 2019-11-08 22:47:50 -08:00
Chris Lu
84c503c6a7 adjust ec reading log level 2019-11-08 22:41:02 -08:00
Chris Lu
6cc9e0d788 volume: fix ec shard reading 
fix https://github.com/chrislusf/seaweedfs/issues/1106
 2019-11-08 22:41:02 -08:00
zhangsong
1dd101f782 use read lock to avoid io hang during heartbeat 2019-11-08 20:27:55 +08:00
stlpmo
1c8bed3810 delete the var etcdClient and comments 
refactor the code
add sequencer cmd-line
delete nerver used codes
 2019-11-05 15:45:25 +08:00
stlpmo
364f7200ad Create etcd_sequencer.go 
the 1st version
 2019-11-04 16:36:06 +08:00
Lei Liu
1294999d8b return error when garbageThreshold is not a valid float number 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-31 20:43:20 +08:00
Lei Liu
1bcef02a6c fix dir/lookup and col/delete api 
1, Fix Layouts first letter capitalized
2, Return http 204 when delete a collection

Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-31 16:55:45 +08:00
Lei Liu
f2f90436ef fix leader master /dir/lookup api 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-30 16:38:40 +08:00
Chris Lu
db30a46050 skip memory mapped file when compacting 2019-10-29 23:34:38 -07:00
Chris Lu
5b950c735e Merge branch 'master' into refactoring_dat_backend 2019-10-29 23:18:41 -07:00
Chris Lu
57e441d67b fix compaction logic 2019-10-29 23:18:01 -07:00
Chris Lu
b7156291a8 adjust variable names 2019-10-29 23:16:43 -07:00
Chris Lu
4b5ba4927f adjust NewDiskFile() paramaters 2019-10-29 22:37:36 -07:00
Lei Liu
c2884cace2 misc updated 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-29 21:28:28 +08:00
Chris Lu
cd7ce720c7 fix test 2019-10-29 00:37:15 -07:00
Chris Lu
19b6a16003 changed from os.file to backend.DataStorageBackend 2019-10-29 00:35:16 -07:00
Chris Lu
eb2172f63f FUSE: add os specific mount options 
related to https://github.com/chrislusf/seaweedfs/issues/1094
 2019-10-27 09:12:10 -07:00
Chris Lu
71eb8efd20 fix error logs 2019-10-25 09:44:58 -07:00
Chris Lu
55a2221eba master: add logging for volume server disconnection 
add logging for https://github.com/chrislusf/seaweedfs/issues/1093
 2019-10-25 09:35:19 -07:00
Chris Lu
69de05f6cb go fmt 2019-10-25 07:45:12 -07:00
Chris Lu
288c45a690 fix iitial filer url 2019-10-25 07:44:37 -07:00
Chris Lu
05fe7a2366
Merge pull request #1092 from binbinshi/master 
fix: weed shell filer config is constant
 2019-10-25 07:08:59 -07:00
shibinbin
fe1f469978 fix : change default filer.port to 8888 2019-10-25 17:39:29 +08:00
shibinbin
3464f21b5c fix: weed shell add filer runtime attributes 2019-10-25 17:34:03 +08:00
Chris Lu
7a51a9a582 refactor: simplifying to ReadAt() and WriteAt() 2019-10-25 00:11:25 -07:00
Chris Lu
46ed2ca902 refactoring 2019-10-24 23:41:32 -07:00
Chris Lu
9573c0f4b3 filer: filer.copy cleanup in case of failed uploads 2019-10-24 10:16:01 -07:00
Chris Lu
1b71608b3c
Merge pull request #1091 from divinerapier/fix/abused-404 
fix abused 404 status code
 2019-10-24 09:35:54 -07:00
Chris Lu
5d78871487 filer: parallize weed filer.copy for single file 2019-10-24 07:26:23 -07:00
divinerapier
f1ee6b7a54
fix abused 404 status code 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-10-24 16:51:26 +08:00
Chris Lu
253dd20bc6 go fmt 2019-10-23 22:25:59 -07:00
Chris Lu
f937933b3d refactoring 2019-10-23 22:25:53 -07:00
Chris Lu
c358f96aaa volume: redirection follow public url 
fix https://github.com/chrislusf/seaweedfs/issues/1089
 2019-10-23 22:11:42 -07:00
Chris Lu
c0776aa40f trying to compile 2019-10-23 01:27:08 -07:00
Chris Lu
23240efa63 filer: add tikv support 2019-10-23 00:31:31 -07:00
Chris Lu
3c865ee39b fix missing needle header read 2019-10-22 00:57:47 -07:00
Chris Lu
c71fddee2b go fmt 2019-10-22 00:50:50 -07:00
Chris Lu
fc412e428b refactor ScanVolumeFileFrom() 2019-10-22 00:50:30 -07:00
Chris Lu
c9a183eb69 refactor memory map related code 2019-10-22 00:49:42 -07:00
Chris Lu
fec07c829d go fmt 2019-10-21 23:03:48 -07:00
Chris Lu
faec9076a4 adjust parameter names 2019-10-21 22:57:01 -07:00
Chris Lu
d16c450682 fix unreachable code 2019-10-21 22:36:41 -07:00
Chris Lu
c262526d8a
Merge pull request #1087 from joeslay/master 
Storing files in in-memory collections on windows
 2019-10-21 22:30:35 -07:00
j.laycock
2e2fe00dbd Comma, no space? 2019-10-18 11:31:25 +01:00
j.laycock
248f3be6e3 using a space instead of a comma to hopefully fix the build! 2019-10-18 11:23:02 +01:00
j.laycock
2c455841ea Make volumeCreate more consistent between all 3 implementations. 2019-10-18 11:01:45 +01:00
j.laycock
b0ddad6889 Fix volume_create breaking the build 2019-10-18 10:32:07 +01:00
joeslay
d53aee179b
Merge pull request #10 from chrislusf/master 
merge seaweed master
 2019-10-14 16:03:40 +01:00
zhangsong
65f1095a54 fix benchmark cmd usage info 2019-10-14 20:23:32 +08:00
Chris Lu
50e885da45 1.44 2019-10-11 23:56:58 -07:00
Chris Lu
7a707617e5 follow the redirected master 
fix https://github.com/chrislusf/seaweedfs/issues/1070
 2019-10-09 21:00:40 -07:00
Chris Lu
87e5a02a99 fix logic error 2019-10-09 00:07:18 -07:00
Chris Lu
939e4b57a8 go fmt 2019-10-09 00:03:18 -07:00
Chris Lu
09874f0d16 volume: return error if superblock is not initialized 
fix https://github.com/chrislusf/seaweedfs/issues/1079
 2019-10-09 00:02:50 -07:00
Chris Lu
f8d4b7d1c0 support basic json filtering and selection 2019-10-06 22:35:08 -07:00
Chris Lu
e26670c67a add some basic sql types 
copied some code from vitness
 2019-10-06 22:35:08 -07:00
Chris Lu
cf47f657af scaffold for volume server query feature 2019-10-06 22:35:08 -07:00
joeslay
9d03aa6a38
Merge pull request #8 from chrislusf/master 
merge seaweed master
 2019-10-01 17:55:31 +01:00
j.laycock
eb27c2b037 Make releaseMemory private and return byte array instead, fix other platform compilation issues, reduce in-memory chunk size. 2019-10-01 12:21:44 +01:00
Lei Liu
2cb348a2c2 s3 api: fix listbucket common_prefixes issue 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-09-29 21:15:40 +08:00
Lei Liu
3cc084269c master api: return http 404 when volumeId not exist 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-09-29 14:23:39 +08:00
j.laycock
d5f5acb734 limit locking physical memory to 80% of max physical memory 2019-09-20 12:44:29 +01:00
joeslay
48b093b3d2
Merge pull request #7 from chrislusf/master 
merge seaweed master
 2019-09-19 12:00:05 +01:00
j.laycock
476140fd6b minor change to setProcessWorkingSetSize function 2019-09-19 11:59:00 +01:00
Chris Lu
972e881d48 add more logs 
add more logs
 2019-09-16 09:48:30 -07:00
Chris Lu
20ae559437 add logs 
add logs for https://github.com/chrislusf/seaweedfs/issues/1064
 2019-09-16 08:19:10 -07:00
Chris Lu
ad3efbb197 tweaking data types 2019-09-14 01:21:51 -07:00
Chris Lu
b0e4771135
Merge pull request #1060 from divinerapier/master 
fix: non-thread-safe rand will panic
 2019-09-14 01:06:14 -07:00
divinerapier
bb31462b52
fix: thread unsafe 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-09-13 20:06:02 +08:00
j.laycock
6fc6322c90 Change joeslay paths to chrislusf paths 2019-09-12 14:18:21 +01:00
joeslay
d8c34b032f
Merge pull request #6 from chrislusf/master 
merge seaweed master
 2019-09-12 11:44:09 +01:00
Chris Lu
ae53f63680 filer: recursive deletion optionally ignoring any errors 
fix https://github.com/chrislusf/seaweedfs/issues/1062
 2019-09-11 20:26:20 -07:00
Chris Lu
5e9c65469e volume: skip readonly checking when Destroy() 
fix https://github.com/chrislusf/seaweedfs/issues/1063
 2019-09-11 09:45:09 -07:00
j.laycock
c1a928887a Add some comments/documentation 2019-09-11 10:42:37 +01:00
joeslay
9ea9a5e97a
Merge pull request #5 from chrislusf/master 
merge seaweed master
 2019-09-11 10:25:27 +01:00
j.laycock
15a4c91c28 Remove Finalizer 2019-09-10 17:05:54 +01:00
j.laycock
9c9dff7386 Virtual lock memory pages and Set Process Working set size to hint to windows harder not to write pages to disk as much, add finalize function, minor renaming 2019-09-10 16:18:51 +01:00
Chris Lu
02710f9624 refactoring: remove unused functions 2019-09-09 10:03:41 -07:00
Chris Lu
0f861d23a2 avoid nil needle map 
fix https://github.com/chrislusf/seaweedfs/issues/1061
 2019-09-09 09:48:08 -07:00
joeslay
d757ed2fe3
Merge pull request #4 from chrislusf/master 
merge seaweed master
 2019-09-06 10:49:27 +01:00
Chris Lu
803345e6af remove glide 2019-09-06 00:15:59 -07:00
j.laycock
63acc8c972 rename mem_buffer to mBuffer 2019-09-05 13:46:29 +01:00
j.laycock
44ae041e80 rename more memorymapped variables to memoryMapMaxSizeMB 2019-09-05 13:42:23 +01:00
j.laycock
5885ab67b3 rename volume property MemoryMap to MemoryMapMaxSizeMB 2019-09-05 11:56:48 +01:00
j.laycock
5a6dfb05fc Only use truncate flag when creating non memory map files, rename and use memory map size for reserving max file size 2019-09-05 11:46:12 +01:00
j.laycock
f6146d6ad2 use 0 instead of false for max memory flag 2019-09-05 11:43:18 +01:00
j.laycock
4257582db5 Allocate in 16MB chunks, make creation of memory maps always aligned to 16MB chunks 2019-09-05 11:15:01 +01:00
j.laycock
84f2dc3b53 Remove test code 2019-09-05 11:14:02 +01:00
Chris Lu
20d90dea5a filer: avoid hard-coded upload timeout 2019-09-05 01:26:59 -07:00
Tom Maxwell
4a878c0006 Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve 2019-09-04 15:27:14 +01:00
joeslay
6ee65356e3
Merge pull request #3 from chrislusf/master 
merge seaweed master
 2019-09-04 14:58:04 +01:00
j.laycock
cc756ddd79 Rename End_of_file variable, implement platform indepenent memory_map.go 2019-09-04 14:24:17 +01:00
j.laycock
9aa0859697 Remove inmemory from vaccum stuff 2019-09-03 18:20:45 +01:00
j.laycock
de5a7e60a2 remove temp hack 2019-09-03 18:20:09 +01:00
j.laycock
cbd0a98fa1 Add InMemory to Volume Grow structure 2019-09-03 18:19:02 +01:00
Chris Lu
ee8bcb0c12 remove println 2019-09-03 09:22:56 -07:00
j.laycock
1f01eb78e8 Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct 2019-09-03 17:00:59 +01:00
Tom Maxwell
d637d86d22 Changes to try and pass the URL parameters through - in memory flag not working still 2019-09-03 15:41:28 +01:00
j.laycock
9a459d984b Do not vacuum memory mapped files 2019-09-03 15:05:43 +01:00
Chris Lu
89a0cb2e11 1.43 2019-09-03 01:31:58 -07:00
j.laycock
0123c7a898 Change to max 2GB for now 2019-09-02 17:46:22 +01:00
joeslay
e91139348e
Merge pull request #2 from chrislusf/master 
merge seaweed master
 2019-09-02 11:39:40 +01:00
j.laycock
595a1beff0 Swap imports to use joeslay 2019-09-02 11:28:40 +01:00
Chris Lu
60c9215a00 HCFS: avoid verbose file not found exception stack trace 2019-09-01 21:40:26 -07:00
j.laycock
1e62a2b233 Fix memory not being unmapped 2019-08-30 16:55:33 +01:00
j.laycock
840ccdc35d Refactor to pass memory maps by reference instead of value, fix memory maps not being created properly or written to properly 2019-08-30 12:30:00 +01:00
j.laycock
dc50701e7d Overload createFile function, expose File in memory_map struct, delete from memory_map map on delete 2019-08-30 12:30:00 +01:00
j.laycock
523f3a12b3 use Os.File/uintptr instead of windows.Handle in memory map struct 2019-08-30 12:30:00 +01:00
j.laycock
0e5d3b1a70 Move memory_map_windows to new folder, intercept most of the read and write calls to volume dat files 2019-08-30 12:30:00 +01:00
Chris Lu
58168a8c52 volume: truncate dat files when creating 2019-08-26 13:01:57 -07:00
joeslay
36bde84000
Merge pull request #1 from chrislusf/master 
merge changes from chrislusf/seaweed master
 2019-08-23 17:03:37 +01:00
j.laycock
1e89e719b3 Add windows memory map functions to storage package 2019-08-23 16:19:00 +01:00
Chris Lu
35d471a818
Revert "use std encoding lib" 2019-08-23 07:59:09 -07:00
Chris Lu
fa160e00f3
Merge pull request #1051 from xushuxun/master 
volume: read dat files' last modified time correctly
 2019-08-20 22:25:37 -07:00
xushuxun
9af9f36b89 volume: read dat files' last modified time no matter dat files are catWrite or readonly 2019-08-21 11:26:01 +08:00
divinerapier
e798e4e9c5
use std encoding lib 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-08-19 18:07:54 +08:00
xushuxun
5904d78bd4 weed backup: add ttl and replication parameter 2019-08-16 11:05:22 +08:00
Chris Lu
f9c1cd7370 avoid dead lock from dataFileAccessLock 
fix https://github.com/chrislusf/seaweedfs/issues/1043
 2019-08-15 09:24:54 -07:00
xushuxun
46ae35cd7e wdclient: clear client cache when failed to connect to master leader 2019-08-15 15:13:16 +08:00
Chris Lu
d829df4f59 volume: protect against nil needle map 
fix @mastak reported nil problem in https://github.com/chrislusf/seaweedfs/issues/1037
 2019-08-14 01:08:01 -07:00
Chris Lu
e40634e6b4 volume: fail the volume deletion if compaction is in progress 
fix https://github.com/chrislusf/seaweedfs/issues/1035
 2019-08-12 00:53:50 -07:00
Chris Lu
6f75df8660 server: fix starting default master list 2019-08-10 03:10:03 -07:00
Chris Lu
0dc43ecdca skip writing content if not modified 
related to https://github.com/chrislusf/seaweedfs/issues/1036
 2019-08-09 14:38:54 -07:00
Chris Lu
b9f6236f8a filer: fix typo 
fix https://github.com/chrislusf/seaweedfs/issues/1031
 2019-08-06 00:42:19 -07:00
Chris Lu
c79274f7eb notes on supported meta data store 2019-08-05 16:45:21 -07:00
Feng Ye
fc0498798b Add etcd filestore 2019-08-01 10:16:45 +08:00
Chris Lu
ca2b81a944 fix test error 2019-07-31 02:09:13 -07:00
Chris Lu
19360ab10a master: redirect clients to the new leader 2019-07-31 01:54:42 -07:00
Chris Lu
8afd8d35b3 master: followers can also lookup and redirect 
improve scalability
 2019-07-28 03:58:13 -07:00
Chris Lu
2c6cf72e73 refactoring 2019-07-28 01:55:05 -07:00
Chris Lu
fb90d63589 shell: Added minor changes to fs.meta.save 
* added an option to set the output file name
* added an option to see verbose output, default to non-verbose mode

related to: https://github.com/chrislusf/seaweedfs/issues/1019
 2019-07-26 09:35:22 -07:00
divinerapier
9e3b77c3ad avoid double warpping 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-07-24 16:41:45 +08:00
Chris Lu
344ce90315 remove weedfuse 2019-07-24 00:42:12 -07:00
Chris Lu
5956dfd08d mount: add umask option 
related to https://github.com/chrislusf/seaweedfs/issues/978
 2019-07-24 00:03:05 -07:00
Chris Lu
3f851feb59 1.42 2019-07-21 21:52:48 -07:00
Chris Lu
862c2cb8e6 go fmt 2019-07-21 21:51:38 -07:00
Chris Lu
79762385bd master: ensure only one exclusive vacuum process 
fix https://github.com/chrislusf/seaweedfs/issues/1011
 2019-07-21 21:49:10 -07:00
Chris Lu
f3b99cbfe5 fix wrong alignment fix 2019-07-21 13:50:42 -07:00
Chris Lu
644c5c27b4 refactoring 2019-07-21 13:50:24 -07:00
Chris Lu
33b3366043 volume: avoid compacting wait group 2019-07-21 13:49:59 -07:00
Chris Lu
4b15c8f0c4 volume: lock writables changes 2019-07-21 13:49:09 -07:00
Chris Lu
fd16adbde1 stricter checking before writes 2019-07-21 11:21:30 -07:00
Chris Lu
5a135fef87 purge old code 2019-07-21 10:45:36 -07:00
Chris Lu
c54d9221b9 write requests also checks cookie if overwrites 
protect against edge cases, avoid https://github.com/chrislusf/seaweedfs/issues/1014
 2019-07-17 23:57:34 -07:00
Chris Lu
898d943b25 refactoring 2019-07-17 23:43:48 -07:00
Chris Lu
0264a7f50a set the max file key when ever possible 2019-07-17 23:23:01 -07:00
Chris Lu
c33f423955 stop early if compaction fails 
fix https://github.com/chrislusf/seaweedfs/issues/1015
 2019-07-17 23:22:01 -07:00
Chris Lu
aff911c00d skip all azuresb 2019-07-17 01:24:20 -07:00
Chris Lu
0ea98dc6a2 disable azure due to api changes 2019-07-17 00:39:36 -07:00
Chris Lu
a7b1b23c58 fix wrong volume count 
fix https://github.com/chrislusf/seaweedfs/issues/1013
 2019-07-17 00:03:15 -07:00
Chris Lu
9e47c4749a shell: add fs.mv 
fix https://github.com/chrislusf/seaweedfs/issues/954
 2019-07-11 09:29:04 -07:00
Chris Lu
1d1b355f9e filer: set file name when uploading to a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1005
 2019-07-11 04:20:03 -07:00
Chris Lu
61b7a650f4 filer.copy default to 32MB chunks 2019-07-10 21:48:27 -07:00
Chris Lu
476a7c269c detect a filer path is a file or a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1004
 2019-07-10 12:11:18 -07:00
Chris Lu
1e2127e261 s3: adding Location for multipart upload 
fix https://github.com/chrislusf/seaweedfs/issues/1002
 2019-07-08 12:37:45 -07:00
Chris Lu
62843991f2 s3: adjust object key 2019-07-08 12:37:20 -07:00
Chris Lu
cf2804eebd 1.41 2019-07-07 20:40:20 -07:00
Chris Lu
7872fc38ea filer leveldb2: correct directory listing 
fix https://github.com/chrislusf/seaweedfs/issues/1001
 2019-07-07 20:37:48 -07:00
Chris Lu
fe2882ce7d fix test 2019-07-02 21:42:06 -07:00
Chris Lu
36250154cc remove println 2019-07-02 21:33:23 -07:00
Chris Lu
4d579f7f57 adjust 2019-07-02 21:28:51 -07:00
Chris Lu
23d36dfbaf 1.40 
enable azure replication
 2019-07-02 21:27:05 -07:00
Chris Lu
dcc331cf5a parallelize leveldb 2019-07-02 21:25:53 -07:00
Chris Lu
88fed85bba temp skipping azure 
can not release due to https://github.com/Azure/azure-pipeline-go/issues/23
 2019-06-30 23:17:45 -07:00
Chris Lu
ce5aab66e9 add versions 2019-06-30 22:43:10 -07:00
Chris Lu
36022f9416 1.39 2019-06-30 00:50:17 -07:00
Chris Lu
15ecf4e3bf filer: default to leveldb2 2019-06-30 00:44:57 -07:00
Chris Lu
ee933cd859 filer: add metrics for go memory, gc stats 2019-06-29 03:02:28 -07:00
Chris Lu
92c7f7e069 fix compilation error 2019-06-28 11:19:08 -07:00
Chris Lu
9f3f2f7c79 protect locations slice 
fix https://github.com/chrislusf/seaweedfs/issues/995
 2019-06-28 09:47:29 -07:00
Chris Lu
327336ecf3 filer: avoid concurrent modification to result slice 
fix https://github.com/chrislusf/seaweedfs/issues/972
 2019-06-28 01:12:41 -07:00
Chris Lu
a3d1296ed9 go fmt 2019-06-27 12:18:59 -07:00
Chris Lu
ec75b2d761 volume: fix bug with 8000GB version if using in memory index 
fix https://github.com/chrislusf/seaweedfs/issues/994
 2019-06-27 12:18:45 -07:00
Chris Lu
6883f9e322 mark volume readonly before ec encoding 2019-06-26 23:02:22 -07:00
Chris Lu
c2960c7a86 defaulting file ttl to volume ttl 
fix https://github.com/chrislusf/seaweedfs/issues/992
 2019-06-26 11:18:23 -07:00
Lei Liu
1d9b75b536
weed.go: remove unused parameter 
Signed-off-by: Lei Liu <liul.stone@gmail.com>
 2019-06-26 10:46:32 +08:00
Chris Lu
359563e95a refactoring 2019-06-25 09:49:27 -07:00
Chris Lu
fdff927b37 1.38 2019-06-24 00:36:46 -07:00
Chris Lu
a8b4fe6eae adjust help text 2019-06-24 00:26:03 -07:00
Chris Lu
c777102da9 go fmt 2019-06-23 15:30:16 -07:00
Chris Lu
d5560f2705 simplify metrics settings 2019-06-23 15:29:49 -07:00
Chris Lu
6f8b335007 fix duplicated flags 2019-06-23 03:11:21 -07:00
Chris Lu
f16375621f big refactoring 2019-06-23 03:08:27 -07:00
Chris Lu
6c01fb6d2d go fmt 2019-06-23 01:57:51 -07:00
Chris Lu
8da5d5b094 filer.copy: use filer settings, avoid unnecessary command line options 
fix https://github.com/chrislusf/seaweedfs/issues/968
 2019-06-23 01:57:35 -07:00
Chris Lu
d1cd8f8c5b add metrics, refactoring 2019-06-22 22:53:52 -07:00
Chris Lu
a14bd31f5b go fmt 2019-06-22 20:05:25 -07:00
Chris Lu
3fa1f150d9 refactoring 2019-06-22 20:04:56 -07:00
Chris Lu
1babec00e7 check deleted chunks faster 2019-06-22 13:22:22 -07:00
Chris Lu
cd45ab072a fix compilation error 2019-06-22 12:30:08 -07:00
Chris Lu
c369e5a13b add metrics for filer store 2019-06-22 12:23:25 -07:00
Chris Lu
6bc3dee5b3 refactoring 2019-06-22 10:56:54 -07:00
Chris Lu
95ad56d99d 1.37 2019-06-21 20:56:34 -07:00
Chris Lu
a111f26fe6 avoid nil 
fix https://github.com/chrislusf/seaweedfs/issues/988
 2019-06-21 20:56:27 -07:00
Chris Lu
308102f023 fix cassandra creation problem 
fix https://github.com/chrislusf/seaweedfs/issues/986
 2019-06-21 17:10:38 -07:00
Chris Lu
fe4eb6bffa 1.36 2019-06-21 13:11:40 -07:00
Chris Lu
059ef879a8 fix issue 986 
fix issue 986
 2019-06-21 13:06:04 -07:00
Chris Lu
ff97acae62 more logs 2019-06-21 12:14:40 -07:00
Chris Lu
d0ce5b020b logging 2019-06-21 12:05:00 -07:00
Chris Lu
78994024ff adjust logging 2019-06-21 11:46:12 -07:00
Chris Lu
f88a8bda7b ec deletion works 2019-06-21 01:14:10 -07:00
Chris Lu
613a2e8060 delete garbage only when successful 2019-06-20 23:46:00 -07:00
Chris Lu
11be0b5e91 more logging 2019-06-20 23:45:30 -07:00
Chris Lu
4b0f084b5d delete chunks only when file writing is successful 2019-06-20 21:58:35 -07:00
Chris Lu
eebc44cce3 refactoring 2019-06-20 21:57:17 -07:00
Chris Lu
3106065b58 fix startup problem with metrics 2019-06-20 09:56:49 -07:00
Chris Lu
c74dc2b306 go fmt 2019-06-20 00:55:52 -07:00
Chris Lu
e63317fb08 ec deletion code complete, not tested yet 2019-06-20 00:55:30 -07:00
Chris Lu
4cea8aefd0 add grpc VolumeEcBlobDelete 2019-06-20 00:17:11 -07:00
Chris Lu
856da7aae2 ec volume support deletes 2019-06-19 22:57:14 -07:00
Chris Lu
115558e5f5 adjust counters 2019-06-17 21:02:50 -07:00
Chris Lu
d8ed73926d volume servers get metrics address and interval from the master 2019-06-17 14:51:47 -07:00
Chris Lu
68d1bef236 instance format 2019-06-16 21:57:32 -07:00
Chris Lu
935639b908 metrics: disk size for <collection, normal/EC> 2019-06-16 21:56:41 -07:00
Chris Lu
0fdb1e705d collect volume disk usage metrics 2019-06-16 02:44:20 -07:00
Chris Lu
289fd7eb39 count number of volumes and ec shards 2019-06-16 02:24:15 -07:00
Chris Lu
ff6a6dd11e refactoring 2019-06-15 21:46:55 -07:00
Chris Lu
8b43679ae3 refactoring 2019-06-15 12:21:44 -07:00
Chris Lu
5336008dcd refactoring 2019-06-14 13:06:01 -07:00
Chris Lu
d67189921f go fmt 2019-06-14 00:55:12 -07:00
Chris Lu
5f6c9825f8 volume server: adds basic metrics 2019-06-14 00:54:56 -07:00
Chris Lu
a11525fe4e filer: adds basic metrics pushing to Prometheus gateway 2019-06-13 02:01:54 -07:00
Chris Lu
be3f1f84a2 1.35 2019-06-10 21:34:06 -07:00
Chris Lu
57092d69ac go fmt 2019-06-10 21:33:32 -07:00
Chris Lu
f9d8bd51ad ec shard balancing 2019-06-10 21:32:56 -07:00
Chris Lu
9d9162ca35 ec.balance: collect dc rack info 2019-06-07 00:25:05 -07:00
Chris Lu
e66bddd84c customizable sleep intervals 2019-06-06 00:39:08 -07:00
Chris Lu
50aa769554 jwt for read access control 2019-06-06 00:29:02 -07:00
Chris Lu
d344e0a035 fix ec related bugs 2019-06-05 23:20:26 -07:00
Chris Lu
450f4733ce report empty volume or ec shards 2019-06-05 13:32:33 -07:00
Chris Lu
7f6fb9ffd0 every 17 minutes instead of seconds 
TODO: make this configurable
 2019-06-05 13:32:08 -07:00
Chris Lu
6b08db65b0 fix shard count reporting 2019-06-05 01:58:37 -07:00
Chris Lu
784141c5e6 adjust script parameter 2019-06-05 01:48:03 -07:00
Chris Lu
ede876cfdb periodic scripts exeuction from leader master 2019-06-05 01:30:24 -07:00
Chris Lu
b9e138713c ec.encode: add -fullPercent option 2019-06-05 00:13:13 -07:00
Chris Lu
0e49add2bc adjust help message 2019-06-05 00:11:37 -07:00
Chris Lu
eaa76f11b7 free volume slots factor in ec shard counts 2019-06-04 23:41:56 -07:00
Chris Lu
ca8a2bb534 go fmt 2019-06-04 22:04:10 -07:00
Chris Lu
2215e81be7 ui add ec shard statuses 2019-06-04 21:52:37 -07:00
Chris Lu
0e52862586 avoid the test that depends on ordering in a different folder 2019-06-04 02:22:32 -07:00
Chris Lu
2f7710a75d debug sporadic test error on travis 2019-06-04 02:09:06 -07:00
Chris Lu
06133ae98f correct count for free ec slots, avoid removing generated shards 2019-06-04 01:49:51 -07:00
Chris Lu
6cea23d091 purge duplicated ec shards 2019-06-04 01:32:36 -07:00
Chris Lu
b05456fe07 able to purge extra ec shard copies 2019-06-03 20:25:02 -07:00
Chris Lu
11cffb3168 fix ec.rebuild bugs 2019-06-03 11:50:54 -07:00
Chris Lu
d85b41b904 fix ec.encode not finding the local ec shards 2019-06-03 10:38:21 -07:00
Chris Lu
7e80b2b882 fix multiple bugs 2019-06-03 02:26:31 -07:00
Chris Lu
55be09996d fix volume balance bug 2019-06-03 00:13:31 -07:00
Chris Lu
9ce6b949bf adjust help message 2019-06-01 10:37:08 -07:00
Chris Lu
d6e2c877fa ec.balance: use specific collection names 2019-06-01 02:00:18 -07:00
Chris Lu
133b772fb5 destroy ec volume if it is empty 2019-06-01 01:51:28 -07:00
Chris Lu
ba18314aab ec shard delete also check ec volumes, in addition to volumes 2019-06-01 01:41:22 -07:00
Chris Lu
f919d0235c ec encode volumes quiet for a period of time 2019-05-31 23:41:17 -07:00
Chris Lu
a72cef3c42 encode by collection 2019-05-31 15:48:40 -07:00
Chris Lu
de1a35acdd conditionally fresh the shard locations 2019-05-31 02:22:12 -07:00
Chris Lu
47f1901843 ask for the ec volume version 2019-05-31 00:58:51 -07:00
Chris Lu
689930f092 forget shards that are broken 2019-05-31 00:19:13 -07:00
Chris Lu
aaab2c25f5 resolve concurrent modification problem 
n and err can be modified concurrently
 2019-05-30 13:59:07 -07:00
Chris Lu
f53024d79d delete ec collection 2019-05-30 10:40:22 -07:00
Chris Lu
40ca2f2903 add collection.delete 2019-05-30 09:47:54 -07:00
Chris Lu
1d111d6ce8 collection list normal and ec volumes 2019-05-30 09:27:23 -07:00
Chris Lu
e941d0c2f4 list ec collections also 2019-05-30 09:17:58 -07:00
Chris Lu
866197eee3 print out the ec balancing plan 2019-05-30 01:38:59 -07:00
Chris Lu
5c6c757619 clean up test files 2019-05-29 00:34:30 -07:00
Chris Lu
5dd67f9acf reading by recover from other shards 2019-05-28 23:48:39 -07:00
Chris Lu
3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu
302d9fbc6d adjust error message for file not found 2019-05-28 14:27:21 -07:00
Chris Lu
4f76342cbc WIP 
no errors, but not returning file content
* the interval needs to use actual file zie
* need to read the actual version instead of the current version
 2019-05-28 00:51:01 -07:00
Chris Lu
2858a954b3 read ec volume shards locations from master 2019-05-28 00:13:13 -07:00
Chris Lu
713596e781 caching ec shard locations 2019-05-27 22:54:58 -07:00
Chris Lu
3a8c1055a2 refactoring ecx to ecVolume 2019-05-27 22:00:36 -07:00
Chris Lu
217cde0a3b refactoring 2019-05-27 21:40:51 -07:00
Chris Lu
03b9291e5d volum server passes grpc option and master location to store 2019-05-27 21:22:23 -07:00
Chris Lu
ac1ba3b667 remove comments 2019-05-27 21:21:52 -07:00
Chris Lu
b4b407e403 add grpc ec shard read 2019-05-27 11:59:03 -07:00
Chris Lu
a4f3d82c57 convert needle id to ec intervals to read from 2019-05-27 01:29:46 -07:00
Chris Lu
a463759edf adjust error message 2019-05-26 01:14:42 -07:00
Chris Lu
92f9c7b1aa adjust ec shard status on disconnect 2019-05-26 01:05:08 -07:00
Chris Lu
ac14cd5ad0 wdclient need to have ec shard info on initial dump 2019-05-26 01:01:04 -07:00
Chris Lu
b58e25e588 broadcast vid->location map to master clients for ec chards 2019-05-26 00:49:15 -07:00
Chris Lu
af67d99ca4 incrementally update master ec shards state 2019-05-26 00:21:17 -07:00
Chris Lu
db94a41f9e mount/unmount ec shards 2019-05-25 23:23:19 -07:00
Chris Lu
41e8ae61f8 generate, balance, delete copied shards, delete old volume 2019-05-25 14:02:06 -07:00
Chris Lu
f0e6574d5e allocate ec shards to volume servers 2019-05-25 02:02:44 -07:00
Chris Lu
6f4b09b6a4 pb shard info uses ShardBits instead one message for one shard 2019-05-24 13:28:44 -07:00
Chris Lu
228850d588 shard id starts from zero 2019-05-24 11:52:23 -07:00
Chris Lu
24b8ad9a14 push the ec shard info to master 2019-05-23 23:47:49 -07:00
Chris Lu
8838176d82 register ec shards to topology 2019-05-23 23:34:29 -07:00
Chris Lu
8a96445f40 register ec shards to each data node 2019-05-23 22:51:18 -07:00
Chris Lu
4659d80035 prepare to register ec shard info in master 2019-05-23 00:42:28 -07:00
Chris Lu
e913ee380a add comments 2019-05-23 00:04:24 -07:00
Chris Lu
7180520889 ec shard info can be queried via VolumeList() 2019-05-22 22:44:28 -07:00
Chris Lu
17ac1290c0 volume: load ec shards during heartbeats to master 2019-05-21 22:41:20 -07:00
Chris Lu
54b835e1ae filer: redis cluster add option for password 
fix https://github.com/chrislusf/seaweedfs/issues/971
 2019-05-20 09:00:30 -07:00
Chris Lu
fbbc74abb4 adds VolumeEcGenerateSlices, VolumeEcCopy 2019-05-20 00:53:17 -07:00
Chris Lu
ae499fd5aa Merge branch 'master' into erasure_coding 2019-05-19 21:38:18 -07:00
Chris Lu
698dea779c refactoring 2019-05-19 21:37:49 -07:00
Chris Lu
693d6c0f1b clean up tests 2019-05-19 21:16:01 -07:00
Chris Lu
0094ca3f8a mark the webdav as unstable 2019-05-19 21:08:26 -07:00
Chris Lu
6386a3174b able to validate by randomly selected ec files 2019-05-19 14:24:33 -07:00
Chris Lu
7c2c60c376 add locating data inside the ec files 2019-05-19 03:01:58 -07:00
Chris Lu
87f63b9c08 generate ec01~ec14, generate ecx file with sorted needle values 2019-05-18 22:46:24 -07:00
Chris Lu
12dc6608f0 Merge branch 'master' into erasure_coding 2019-05-18 11:16:07 -07:00
Chris Lu
f2c4c888f6 1.34 2019-05-17 22:53:30 -07:00
Chris Lu
8123a488cd filer: optimize leveldb parameters 2019-05-17 18:14:25 -07:00
Chris Lu
76cf57a0e1 volume: optimzation for leveldb 2019-05-17 17:34:03 -07:00
Chris Lu
0fe286a6ca filer: optimization for leveldb, add leveldb2 
large filer db may see high CPU and disk usage due to background compaction
 2019-05-17 17:33:49 -07:00
Chris Lu
7e0c3bb732 filer: adjust recommended mysql meta data type to LONGBLOB 
fix https://github.com/chrislusf/seaweedfs/issues/966
 2019-05-17 09:11:08 -07:00
Chris Lu
bf9d490538 fix for tests 2019-05-17 02:28:20 -07:00
Chris Lu
82b0759493 filer: migrating filer store from persisting shorter structured file id instead of a string 2019-05-17 02:03:23 -07:00
Chris Lu
8156958ee9 move function to make travis happy 2019-05-15 10:02:44 -07:00
Chris Lu
0a36f628c6 testing RS coding 2019-05-15 01:02:00 -07:00
Chris Lu
ab22407763 filer ip only use bind ip 2019-05-14 23:43:40 -07:00
Chris Lu
017d0957c3 avoid switching master node if same ip and host name 
fix https://github.com/chrislusf/seaweedfs/issues/955
 2019-05-11 18:42:46 -07:00
Chris Lu
4dfcd2169e weedfuse: adapt to fstab command line pattern 2019-05-11 01:17:55 -07:00
Chris Lu
9a4fb14ea0 weedfuse: daemonize 2019-05-10 17:08:15 -07:00
Chris Lu
1ca1ec906a add weedfuse 2019-05-10 15:03:31 -07:00
Chris Lu
69b7dd398d adjust help message 2019-05-07 14:02:01 -07:00
Chris Lu
d01807ba05 shell: volume.balance add option for all_collections, each_collection 2019-05-06 14:12:19 -07:00
Chris Lu
8ffdd5dcdb writel throttling moving volumes 2019-05-06 13:58:42 -07:00
Chris Lu
cf58fc0e63 refactor: extract out the write throttler 2019-05-06 13:56:08 -07:00
Chris Lu
4e42e7b5e7 shell: volume.balance add dataCenter filter 2019-05-06 13:30:14 -07:00
Chris Lu
4c392ca762 use ip.bind to listen for grpc calls 
help to avoid https://github.com/chrislusf/seaweedfs/issues/952
 2019-05-06 08:10:18 -07:00
Chris Lu
8656102127 shell: volume.balance runs for each collection, and for ALL at the end 2019-05-05 22:28:14 -07:00
Chris Lu
f08bbb72de refactoring 2019-05-05 22:21:28 -07:00
Chris Lu
0df6346611 add error checking 2019-05-05 21:58:46 -07:00
Chris Lu
3ff92037cc shell: add volume.balance command 2019-05-05 21:17:23 -07:00
Chris Lu
3f2a30bd05 shell: volume.list output sorted voume list 2019-05-05 20:23:50 -07:00
Chris Lu
46a675ecc2 align memory to avoid nil with atomic operations 2019-05-04 21:33:05 -07:00
Chris Lu
177f3add81 adjust memory layout to avoid possible nil pointer 2019-05-04 08:47:11 -07:00
Chris Lu
25941e0500 master: add jwt expires_after_seconds 2019-05-04 08:42:25 -07:00
Chris Lu
bd8af92b54 remove unused code 2019-05-04 08:20:22 -07:00
Chris Lu
4b27f892e6 remove 128 bit needle id build option 
fix https://github.com/chrislusf/seaweedfs/issues/949
 2019-05-03 22:39:00 -07:00
Chris Lu
3fbe22b203 better error handling for nil raftServer 
addressing https://github.com/chrislusf/seaweedfs/issues/717
 2019-05-03 22:33:08 -07:00
Chris Lu
5681208f2d refactor: remove logger 2019-05-03 22:21:06 -07:00
Chris Lu
8c982db25d 1.33 2019-05-03 17:27:50 -07:00
Chris Lu
b335f81a4f volume: add option to limit compaction speed 2019-05-03 17:22:39 -07:00
Chris Lu
f0f981e7c8 webdav: fix directory renaming 2019-05-03 14:12:51 -07:00
Chris Lu
bfbecd7253 webdav: fix directory creation 2019-05-03 13:13:11 -07:00
Chris Lu
9d3c945b00
Merge pull request #950 from chrislusf/webdav 
add WebDav
 2019-05-03 01:25:10 -07:00
Chris Lu
7064b2ea48 webdav: can write now 2019-05-03 00:55:52 -07:00
Chris Lu
b30c14b631 webdav: can read now 2019-05-03 00:24:35 -07:00
Chris Lu
47f14775d7 scaffolding, but seems hard to satisfy mac, windows, and different os versions 
https://www.reddit.com/r/golang/comments/3wrxo3/webdav_server_that_provide_filesystem_using/
 2019-05-02 14:22:10 -07:00
Wine93
32f93fb09e style: go fmt for all 2019-04-30 03:22:19 +00:00
Chris Lu
82e06a780b mount: allow nonempty fuse mount to work with Samba 
possibly fix https://github.com/chrislusf/seaweedfs/issues/936
 2019-04-29 10:42:09 -07:00
Chris Lu
108d0fb08d adjust log level 2019-04-26 09:32:37 -07:00
Chris Lu
575926cdec minor 2019-04-26 09:32:37 -07:00
Chris Lu
630cf1e33e fix 2019-04-24 12:31:52 -07:00
Chris Lu
ea4592161c fix 2019-04-24 12:31:14 -07:00
Chris Lu
c5a931b22c logging 2019-04-24 12:30:36 -07:00
Chris Lu
9beea63960 memory alignment 
fix https://github.com/chrislusf/seaweedfs/issues/939
 2019-04-24 09:23:19 -07:00
Chris Lu
9774ede3cf adjust usage description 2019-04-24 00:25:20 -07:00
Chris Lu
b04d7e3ac0 weed server: also optionally start S3 gateway 2019-04-24 00:18:01 -07:00
Chris Lu
dabc9c9a89 filer: listen on specified ip address 2019-04-24 00:01:39 -07:00
Chris Lu
21fe5bde7c filer: remove customizable grpc port, fixed to filer port + 10000 2019-04-23 23:58:52 -07:00
Chris Lu
067b935763 report unknown commands 2019-04-23 22:29:36 -07:00
Chris Lu
53e2e73de2 fix metrics 
fix https://github.com/chrislusf/seaweedfs/issues/935
 2019-04-22 12:32:10 -07:00
Chris Lu
0c96cfdb2e minor 2019-04-21 15:50:51 -07:00
Chris Lu
00b6f653fa shell: add fs.cat 2019-04-21 15:43:43 -07:00
Chris Lu
59d532a8c3 1.31 2019-04-21 14:09:50 -07:00
Chris Lu
5c411f3e5f minor 2019-04-21 13:33:32 -07:00
Chris Lu
440111a349 volume: support http status 304 for the same file id 2019-04-21 13:33:23 -07:00
Chris Lu
ebe94be1a1 maintain layout correctness when changing volumes 2019-04-21 13:32:36 -07:00
Chris Lu
0302b9496c volume: also check cookie for possible duplicates 2019-04-21 13:31:45 -07:00
Chris Lu
132921ad41 adjust log or error 2019-04-21 10:14:17 -07:00
Chris Lu
316bd27f75 add remove volumes with version info 2019-04-20 23:53:37 -07:00
Chris Lu
64a9a0e104 shell: fix volume.copy 2019-04-20 20:48:07 -07:00
Chris Lu
5f3d0e33a1 shell: add commands volume.copy volume.delete volume.mount volume.unmount 2019-04-20 20:31:35 -07:00
Chris Lu
d06d3c57d4 add notes 2019-04-20 20:12:07 -07:00
Chris Lu
2ffe98443b refactoring 2019-04-20 12:05:28 -07:00
Chris Lu
6fc1f53018 shell: add command volume.move 2019-04-20 11:35:23 -07:00
stlpmo-jn
2200ea9cb9 add function ParseFileIdFromString 2019-04-20 18:39:06 +08:00
Chris Lu
3b3651dea3 volume: atomic copying file, adds version and stopOffset 2019-04-19 12:29:49 -07:00
Chris Lu
730a032137 filer mysql: adjust scaffolding instructions 2019-04-19 09:55:46 -07:00
Chris Lu
0be2d51c96 read volume lastAppendAtNs when loading a volume 2019-04-19 01:56:38 -07:00
Chris Lu
ac2727853f fix needle map entry size 2019-04-19 00:39:34 -07:00
Chris Lu
e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu
33c92b819a refactoring 2019-04-18 21:17:43 -07:00
Chris Lu
a2d34d4802 filer.copy: add adjustable compression level 2019-04-18 20:21:28 -07:00
Chris Lu
fa176fe80f volume tailing chunks large files 2019-04-18 19:22:13 -07:00
Chris Lu
3dce1016cb add volume tailer 2019-04-18 11:05:02 -07:00
Chris Lu
b142f9f1d5 go fmt 2019-04-18 00:19:18 -07:00
Chris Lu
b09e8dbb37 add VolumeStreamFollow, but not used yet 2019-04-18 00:18:29 -07:00
Chris Lu
13ad5c1966 refactoring 2019-04-17 22:04:49 -07:00
Chris Lu
3b9446606a adjust log level 2019-04-16 23:31:44 -07:00
Chris Lu
07091c44cd lock btree writes 
fix https://github.com/chrislusf/seaweedfs/issues/930
 2019-04-16 19:48:07 -07:00
Chris Lu
4393b99332 add notes 2019-04-16 13:57:45 -07:00
Chris Lu
338e6d60a5 refactor: prepare for snapshotting 2019-04-16 09:55:37 -07:00
Chris Lu
af49aea0c6 weed shell: fs.meta.save adjusts meta data file name 2019-04-16 09:40:27 -07:00
Chris Lu
8ea1ee6dfa weed shell: add fs.meta.notify, removing filer.export 2019-04-16 01:58:28 -07:00
Chris Lu
014906ec79 weed shell: add command fs.meta.load to restore meta data 2019-04-16 01:37:11 -07:00
Chris Lu
79c2cca9c1 better error message 2019-04-16 01:15:30 -07:00
Chris Lu
d35023c713 weed shell: add command fs.meta.save 2019-04-16 01:06:32 -07:00
Chris Lu
b3b42bc947 replicate need to include new entry path 2019-04-16 00:44:31 -07:00
Chris Lu
967e108b9a increase max idle conn per hosts 2019-04-15 22:00:50 -07:00
Chris Lu
f567777d64 volume: fix needle checking for deleted needles 2019-04-15 21:58:43 -07:00
Chris Lu
bd76ad0ff1 Revert "Revert "volume: close request body if any parsing error"" 
This reverts commit 359b7c1650.
 2019-04-15 21:43:28 -07:00
Chris Lu
359b7c1650 Revert "volume: close request body if any parsing error" 
This reverts commit 5996656141.
 2019-04-15 21:35:12 -07:00
Chris Lu
5996656141 volume: close request body if any parsing error 2019-04-15 21:11:45 -07:00
Chris Lu
6e116b3d67 volume: reset leader if error happens 2019-04-15 09:09:46 -07:00
Chris Lu
e85048bcdc http exhaust and close response body 2019-04-14 23:28:24 -07:00
Chris Lu
3e8a3a8fec fix race detector found problems 2019-04-14 23:00:37 -07:00
Chris Lu
2e03f3adef filer: rename -port.public to -port.readonly 2019-04-11 20:42:55 -07:00
Chris Lu
59a9e15733 1.30 2019-04-11 19:40:36 -07:00
Chris Lu
e11e127d91 close grpc connections during errors 2019-04-11 09:27:08 -07:00
Wine93
361912224d typo: remove blank 2019-04-11 09:18:53 +00:00
Chris Lu
0e33272f56 small refactoring 2019-04-10 23:39:53 -07:00
Chris Lu
cb2061f86a
Merge pull request #925 from stlpmo-jn/checkUnhealthReplication 
the implement of repair the error replications of the volume
 2019-04-10 23:37:12 -07:00
Chris Lu
c0d6cc0067 weed mount: mount to the same folder name 2019-04-10 23:25:41 -07:00
stlpmo-jn
c3fa50d3b3 remove the health chekcer, because it's the same as command_volume_fix_replication 2019-04-11 13:40:31 +08:00
Chris Lu
2f76681d62 refactor: adjust grpc API name 2019-04-10 21:41:17 -07:00
stlpmo-jn
f2031884f0 fix bug : CI build failed 2019-04-11 09:53:31 +08:00
Chris Lu
4df10e50b3 update glide.yaml 2019-04-10 14:27:28 -07:00
stlpmo-jn
c1a0403da2 repair the error replications of the volume 2019-04-10 19:41:55 +08:00
Chris Lu
3555628ad5 weed volume: fix bug, remove boltdb, btree 2019-04-09 10:08:59 -07:00
Chris Lu
8a5ce16e96 weed volume: remove boltdb, btree options, add options for leveldb medium, large 2019-04-09 09:42:06 -07:00
Chris Lu
9924fa3b1a weed volume: disable block cache on leveldb 
saving 8MB for each volume, but at the cost of slower lookup
 2019-04-09 00:24:32 -07:00
Chris Lu
35aba35c07 Revert "weed volume: disable block cache on leveldb" 
This reverts commit a55e708d36.
 2019-04-09 00:22:40 -07:00
Chris Lu
a55e708d36 weed volume: disable block cache on leveldb 
saving 8MB for each volume, but at the cost of slower (offset,size) lookup
 2019-04-09 00:18:32 -07:00
Chris Lu
6d4c44a85c 1.29 2019-04-08 22:41:59 -07:00
Chris Lu
88ba08cb9e add large disk support 2019-04-08 22:32:42 -07:00
Chris Lu
784c5bb73a add build option support 5-byte offset 2019-04-08 22:01:29 -07:00
Chris Lu
7a4b234ea1 divide offset into higher and lower sections 2019-04-08 21:44:06 -07:00
Chris Lu
000ee725fc refactor Offset into a struct of bytes 2019-04-08 19:40:56 -07:00
Chris Lu
a41ba79119
Merge pull request #914 from jba/gocdk-receive 
replication: add GoCDK PubSub support
 2019-04-08 10:31:02 -07:00
Chris Lu
d14b614407 weed filer.copy: use existing file owner and gropu id 2019-04-07 11:31:50 -07:00
Chris Lu
a32abda1a3 adjust compression to optimize for speed 2019-04-07 11:01:28 -07:00
Chris Lu
c731d9be6d weed filer.copy: add option to profile the execution 2019-04-07 09:13:24 -07:00
Chris Lu
174bf1e8b2 more efficient client side gzip compression 2019-04-06 14:14:28 -07:00
Chris Lu
868913aa04 printout statistics 2019-04-06 11:12:35 -07:00
Chris Lu
de7626bd22 weed shell: volume.list add summary statistics 2019-04-06 09:25:29 -07:00
Chris Lu
94302935b2 weed volume: only store compressed data if compression is effective 2019-04-06 00:10:52 -07:00
Chris Lu
b2d92a2992 weed shell: fs.tree fix directory tree listing 2019-04-05 23:42:36 -07:00
Chris Lu
24c020104a weed filer.copy: parallelize the copying to increase throughput 
improvement https://github.com/chrislusf/seaweedfs/issues/915
 2019-04-05 23:35:30 -07:00
Chris Lu
c789b496d8 use cached grpc client 2019-04-05 20:31:58 -07:00
Chris Lu
5808caa2f5 use cached grpc client 2019-04-05 20:26:52 -07:00
Chris Lu
cd6d35aa52 weed shell: fs.tree improvements 2019-04-05 11:40:54 -07:00
Jonathan Amsterdam
72920efc20 added entry to scaffold 2019-04-05 13:43:38 -04:00
Jonathan Amsterdam
a6a5d80401 fix typo 2019-04-05 08:13:32 -04:00
Chris Lu
300b39b5af weed filer.copy: use cached filer grpc connection 
another attemp for https://github.com/chrislusf/seaweedfs/issues/915
 2019-04-05 01:09:06 -07:00
Chris Lu
921231268b weed filer.copy: skip files that can not be opened 
such as socket
 2019-04-05 00:04:00 -07:00
Chris Lu
4f714cef22 default weed filer.copy and weed upload chunk size to 32MB 2019-04-04 23:14:32 -07:00
Chris Lu
c94823f52c set default http idle connection per host 
possible fix https://github.com/chrislusf/seaweedfs/issues/915
 2019-04-04 20:28:40 -07:00
Chris Lu
0bd7ced7c2 fix test 2019-04-04 20:11:59 -07:00
Chris Lu
9cc73f4a9a fix test 2019-04-04 19:34:15 -07:00
Chris Lu
af37b374cb weed shell: fs.cd change current directory 2019-04-04 19:27:51 -07:00
Chris Lu
766396d249 weed master: atomic volume counting 
possible fix for https://github.com/chrislusf/seaweedfs/issues/913
 2019-04-04 19:27:00 -07:00
Jonathan Amsterdam
c0c9a8bad5 replication: add GoCDK PubSub support 2019-04-04 17:22:45 -04:00
Chris Lu
715a38da1e weed shell: add fs.cd, fs.pwd to change to a directory and print current directory 2019-04-03 00:20:00 -07:00
Chris Lu
20dcb44077 fix tests 2019-04-01 23:59:31 -07:00
Chris Lu
beaa2bd71a weed filer: properly delete cached directory entry 
fix https://github.com/chrislusf/seaweedfs/issues/910
 2019-04-01 17:03:04 -07:00
Chris Lu
78b9db34d5 weed s3: generate time format acceptable by aws cli s3 
fix https://github.com/chrislusf/seaweedfs/issues/911
 2019-04-01 16:13:29 -07:00
Chris Lu
2a52e70a41 weed filer: increase default mysql/postgres dir or name length to 65535 2019-04-01 12:37:54 -07:00
Chris Lu
b4c276263f weed filer: mysql/postgres use ReadCommitted transaction isolation level for atomic rename 2019-04-01 11:03:04 -07:00
Chris Lu
c5a3ff1c35 weed s3: multipart upload goes to the right bucket 
fix https://github.com/chrislusf/seaweedfs/issues/908
 2019-03-31 11:10:19 -07:00
Chris Lu
189c890715 weed replicate: replicate atomic rename to other systems 2019-03-31 08:10:47 -07:00
Chris Lu
8c823abe1f 1.28 2019-03-30 23:22:41 -07:00
Chris Lu
78ac2bef3c go fmt 2019-03-30 23:09:16 -07:00
Chris Lu
97406333a5 support atomic renaming for mysql/postgres filer store 2019-03-30 23:08:29 -07:00
Chris Lu
920b4e56aa refactoring 2019-03-29 22:53:35 -07:00
Chris Lu
f60d383947 weed filer.copy: support multiple masters 2019-03-29 22:50:14 -07:00
Chris Lu
487ca51491 weed filer.export: fix pagination error 2019-03-28 00:46:15 -07:00
Chris Lu
07775998a9 some comments 2019-03-28 00:38:30 -07:00
Chris Lu
53b81fcfcd weed shell: add fs.tree 2019-03-28 00:05:04 -07:00
Chris Lu
c7e7b6229f detect mime type 2019-03-27 14:25:18 -07:00
Chris Lu
65757ae2fd weed filer: set content-type consistent with filer store 2019-03-27 11:41:11 -07:00
Chris Lu
37ce4c5269 weed shell: add fs.ls 2019-03-26 12:43:51 -07:00
Chris Lu
7a14cdc90c refactoring, go fmt 2019-03-25 23:18:40 -07:00
Chris Lu
19728fe3f6 remove deprecated code 2019-03-25 23:12:14 -07:00
Chris Lu
df95ce0b6c weed backup: efficient delta backup 
fix https://github.com/chrislusf/seaweedfs/issues/399
 2019-03-25 23:01:53 -07:00
Chris Lu
c913583e2d Merge branch 'master' into add_volume_follow 2019-03-25 21:19:34 -07:00
Chris Lu
91b9a68493 udpate glide 2019-03-25 09:39:54 -07:00
Chris Lu
a32797518b weed master: redirect to leader for default admin UI 
fix https://github.com/chrislusf/seaweedfs/issues/898
 2019-03-25 09:20:05 -07:00
Chris Lu
70815e9124 WIP 2019-03-25 09:16:12 -07:00
Chris Lu
eaa42c3865 1.27 2019-03-23 14:14:40 -07:00
Chris Lu
cd8a3b99bb textual changes 2019-03-23 12:57:35 -07:00
Chris Lu
bd1c0735e0 weed shell: adjust help text format 2019-03-23 11:54:26 -07:00
Chris Lu
6b70b36105 weed shell: add "volume.fix.replication" 2019-03-23 11:34:09 -07:00
Chris Lu
95e0520182 weed volume: add grpc operation to relicate a volume to local 2019-03-23 11:33:34 -07:00
Chris Lu
a3490b600c weed filer, weed master: add option to disable http 2019-03-21 16:00:46 -07:00
Chris Lu
531add52c2 weed master: skip proxied file read in http read only mode 2019-03-21 09:49:04 -07:00
Jonathan Amsterdam
e966033098 Merge branch 'master' of https://github.com/chrislusf/seaweedfs into gocdk 2019-03-20 16:02:38 -04:00
Chris Lu
977b30e992 weed filer: http HEAD response header add ETag 
fix https://github.com/chrislusf/seaweedfs/issues/892
 2019-03-20 10:13:53 -07:00
Jonathan Amsterdam
8db82e2b75 notification: add Go CDK pubsub support 
Add the gocdk_pub_sub package, which supports the Go Cloud Development
Kit pubsub API.

Link in all current providers.

Update the notification scaffold.
 2019-03-20 07:57:58 -04:00
chenwanli
45a52b17fd benchmark: add replication flag 2019-03-20 16:38:11 +08:00
Chris Lu
88ab932f7d refactoring function parameter 2019-03-19 23:01:23 -07:00
Chris Lu
5ae4b963a4 avoid using global rand 2019-03-19 22:20:14 -07:00
Chris Lu
f3d316a846 weed shell: adding file system disk usage fs.du 
fix https://github.com/chrislusf/seaweedfs/issues/889

echo "fs.du http://localhost:8888/some/path" | weed shell
 2019-03-19 21:58:00 -07:00
Chris Lu
cea5c53bf7 fix length checking 
fix https://github.com/chrislusf/seaweedfs/issues/890
 2019-03-19 20:56:27 -07:00
Chris Lu
da871896c3 weed filer: set grpc port to port + 10000 2019-03-19 05:47:41 -07:00
Chris Lu
916b809c08 add error checking 2019-03-19 05:34:43 -07:00
Chris Lu
eb4a54d9fe refactoring 2019-03-19 05:19:37 -07:00
Chris Lu
ece9d13312 volume info collect compact revision number 2019-03-18 09:32:21 -07:00
Chris Lu
104922a3db text wrapping 2019-03-18 00:35:15 -07:00
Chris Lu
44647a46c0 needle scanner read appendAtNs 2019-03-17 23:28:43 -07:00
Chris Lu
8b26d15740 refactoring: simplify function parameter 2019-03-17 22:32:01 -07:00
Chris Lu
2fcc88116e go fmt 2019-03-17 20:27:36 -07:00
Chris Lu
aca653c08b weed shell: list volumes 2019-03-17 20:27:08 -07:00
Chris Lu
22fbbf023b adjust collection list output 2019-03-17 17:31:05 -07:00
Chris Lu
e48267e287 adjust output 2019-03-17 17:28:29 -07:00
Chris Lu
657dd2e6c9 add shell command to list all collections 2019-03-16 13:43:16 -07:00
Chris Lu
b92122b885 fix test 2019-03-16 09:50:57 -07:00
Chris Lu
36b632ebcb fix tests 2019-03-16 09:50:21 -07:00
Chris Lu
42cb05c088 add context.Context 2019-03-15 17:23:02 -07:00
Chris Lu
55bab1b456 add context.Context 2019-03-15 17:20:24 -07:00
Chris Lu
cece860bfd add context to all filer APIs 2019-03-15 15:55:34 -07:00
Chris Lu
55c85f3a66 adding context 2019-03-15 15:26:09 -07:00
Chris Lu
d30600e6d6 fix filer starting under "weed server" when ip is not localhost 
fix https://github.com/chrislusf/seaweedfs/issues/886
 2019-03-15 09:57:33 -07:00
Chris Lu
a4e8aa98a0 weed server: mkdir first, and set the master ip address 
fix https://github.com/chrislusf/seaweedfs/issues/885
 2019-03-15 00:32:42 -07:00
Chris Lu
95ef4513c8 print out memory size 2019-03-13 23:14:40 -07:00
Chris Lu
346541a101 print out per entry memory usage 2019-03-13 23:07:24 -07:00
Chris Lu
99ce10daeb Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-03-11 00:58:17 -07:00
Chris Lu
2ce4dae795 move to 1.26 2019-03-11 00:58:12 -07:00
chenwanli
c668e55e07 Revert "fix https://github.com/chrislusf/seaweedfs/issues/780" 
This reverts commit 31038acccf.
 2019-03-11 11:48:11 +08:00
Chris Lu
6fe071175d mount: add option to disable allow others 
fix https://github.com/chrislusf/seaweedfs/issues/877
 2019-03-10 19:45:40 -07:00
Chris Lu
ad08a52ab6 synchronously upload files 
fix https://github.com/chrislusf/seaweedfs/issues/807
 2019-03-09 13:24:22 -08:00
Chris Lu
4773497d2c add back "/cluster/status" 
fix https://github.com/chrislusf/seaweedfs/issues/870
 2019-03-06 12:10:45 -08:00
Chris Lu
ab32556a51 weed server: fix error coming from moving raft to grpc 
fix https://github.com/chrislusf/seaweedfs/issues/872
 2019-03-06 10:21:49 -08:00
Chris Lu
0898e6c38e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-03-04 13:00:10 -08:00
Chris Lu
d312c55bbe file path supports windows, avoiding back slashes 
fix https://github.com/chrislusf/seaweedfs/issues/868
 2019-03-04 13:00:08 -08:00
Chris Lu
f9dcf56465 adjust log level for loading configurations 2019-03-04 09:02:58 -08:00
chenwanli
f233bbe9c0 Fix https://github.com/chrislusf/seaweedfs/issues/825#issuecomment-469097375 2019-03-04 12:43:43 +08:00
Chris Lu
219b651bc3 jwt check the base file id 
fix https://github.com/chrislusf/seaweedfs/issues/867
 2019-03-03 10:17:44 -08:00
Chris Lu
2812c14520 master: add option to disable http operations 2019-03-02 05:10:05 -08:00
chenwanli
900aad4935 Fix https://github.com/chrislusf/seaweedfs/issues/864 2019-02-28 11:37:44 +08:00
Chris Lu
cb07d15254 add namespace for s3 2019-02-27 00:21:37 -08:00
Chris Lu
70ac2f6ea8 add namespace for ListAllMyBucketsResult 2019-02-26 23:23:04 -08:00
chenwanli
fd27ed7755 Fix https://github.com/chrislusf/seaweedfs/issues/861 2019-02-26 17:12:39 +08:00
Chris Lu
344caf3cd7 1.25 2019-02-25 00:44:48 -08:00
Chris Lu
7bbe24dd28 volume server directly support https 2019-02-25 00:43:36 -08:00
Chris Lu
c892b898a1 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-02-25 00:34:25 -08:00
Chris Lu
0d84e32f56 update library versions 2019-02-25 00:34:21 -08:00
chenwanli
0bc3cdd26a Fix https://github.com/chrislusf/seaweedfs/issues/825 2019-02-25 10:47:41 +08:00
Chris Lu
0a106c1757 updated glide 2019-02-20 09:09:47 -08:00
Chris Lu
e108688990 avoid grpc 5 seconds timeout 
some operations may take longer than 5 seconds.

only keep the timeout for raft operations
 2019-02-20 01:01:01 -08:00
Chris Lu
0fa1be8c4b adjust formatting and comments 2019-02-19 21:10:10 -08:00
Chris Lu
72c2e6bb8f remove unused file 2019-02-19 14:16:19 -08:00
Chris Lu
07af52cb6f raft change from http to grpc 
master grpc port is fixed to http port + 10000
 2019-02-18 22:38:14 -08:00
Chris Lu
448645203a remove unused variables 2019-02-18 20:05:55 -08:00
Chris Lu
d37c3ab7a5 adjust error message 2019-02-18 18:03:27 -08:00
Chris Lu
2442d56671 adjust package name 2019-02-18 18:03:16 -08:00
Chris Lu
9a4dda3011
Merge pull request #855 from chrislusf/add_jwt 
Add jwt
 2019-02-18 15:05:32 -08:00
Chris Lu
a1c7dc3806 avoid changing attributes for mount directory 2019-02-18 12:14:28 -08:00
Chris Lu
77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu
55761ae806 skip nil options 2019-02-16 12:49:58 -08:00
Chris Lu
98a03b38e5 avoid util package depends on security package 2019-02-16 12:23:35 -08:00
Chris Lu
31aa60ce05 Merge branch 'master' into add_jwt 2019-02-16 08:55:38 -08:00
Chris Lu
ad257ae179 simplify isLeader() logic 2019-02-16 08:55:23 -08:00
Chris Lu
712a0e19c8 Merge branch 'master' into add_jwt 2019-02-15 10:01:16 -08:00
Chris Lu
2ec6a679c2 avoid "fchmod failed" during cp 2019-02-15 10:00:27 -08:00
Chris Lu
157c0f7c01 add jwt for writes 2019-02-15 09:59:22 -08:00
Chris Lu
7103c1ab7e go fmt 2019-02-15 00:09:48 -08:00
Chris Lu
74fb237727 benchmark can work in secure mode 2019-02-15 00:09:19 -08:00
Chris Lu
a3b0e39b06 weed mount deletion always goes to weed filer 2019-02-14 07:07:38 -08:00
Chris Lu
215cd27b37 add authorizing fileId write access 
need to secure upload/update/delete for benchmark/filer/mount
need to add secure grpc
 2019-02-14 00:08:20 -08:00
Chris Lu
4ff4a147b2 cleanup security.Secret 2019-02-09 21:56:32 -08:00
Chris Lu
501bd72b1c wip: add security.toml file 2019-02-09 21:07:12 -08:00
Chris Lu
729d71fd4c Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-02-09 12:52:12 -08:00
Chris Lu
8afc632484 raft: use the first master to bootstrap the election 2019-02-09 12:52:09 -08:00
Chris Lu
744abc2690 log raft state changes 2019-02-09 12:51:14 -08:00
Sergey
aa5ccff6d2
fixing of typos 2019-02-06 18:59:15 +05:00
Chris Lu
1334507595 Revert "randomize based on self address" 
This reverts commit 6230eb28a6.
 2019-01-28 12:12:51 -08:00
Chris Lu
6230eb28a6 randomize based on self address 
fix #851
 2019-01-28 11:55:33 -08:00
Chris Lu
221105eea3 Revert "use the first entry to bootstrap master cluster" 
This reverts commit 40c8725ffa.
 2019-01-28 11:46:46 -08:00
Chris Lu
7a493bbefa better way to detect a master isLeader 2019-01-28 10:36:16 -08:00
Chris Lu
40c8725ffa use the first entry to bootstrap master cluster 
fix https://github.com/chrislusf/seaweedfs/issues/851
 2019-01-28 10:35:28 -08:00
Chris Lu
adcfe66034 1.24 2019-01-26 00:15:50 -06:00
Chris Lu
834f414af9 add a timeout 2019-01-26 00:15:42 -06:00
Chris Lu
09471b46f9 fix related tests 2019-01-26 00:10:29 -06:00
Chris Lu
5668ed798d memory needle map mark size to be TombstoneFileSize 
fix https://github.com/chrislusf/seaweedfs/issues/850
 2019-01-26 00:05:44 -06:00
Chris Lu
d3f675abc6 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2019-01-24 22:39:41 -08:00
Chris Lu
e70f740deb set file attributes 
fix https://github.com/chrislusf/seaweedfs/issues/847
 2019-01-24 19:38:04 -08:00
bingoohuang
cf0fc797ac use tempFile.Truncate to reset file size to speedup test 2019-01-23 16:24:51 +08:00
Chris Lu
3f56b12ed4 raft: adding idle connection time out 
another attempt to fix https://github.com/chrislusf/seaweedfs/issues/825
 2019-01-22 09:25:25 -08:00
chenwanli
f47f8be912 Fix block-waiting for leader when there are no volumes 2019-01-22 15:58:37 +08:00
Chris Lu
8ec57e18c7 1.23 2019-01-18 14:27:34 -08:00
Chris Lu
a6bffc58f9 go fmt 2019-01-18 14:15:10 -08:00
Chris Lu
67e2ea72be master add separate grpc port 
due to https://github.com/soheilhy/cmux/issues/64

fix https://github.com/chrislusf/seaweedfs/issues/820
fix https://github.com/chrislusf/seaweedfs/issues/840
fix https://github.com/chrislusf/seaweedfs/issues/841
 2019-01-18 14:14:47 -08:00
Chris Lu
dfae0f4e9d 1.22 2019-01-18 00:25:20 -08:00
Chris Lu
e84dcb0bfe fix timeout issue, but can not reproduce with local built binaries 
fix https://github.com/chrislusf/seaweedfs/issues/841
 2019-01-18 00:24:40 -08:00
Chris Lu
1d103e3ed5 timeout http connections 
possible fix for https://github.com/chrislusf/seaweedfs/issues/825
 2019-01-17 23:38:33 -08:00
Chris Lu
b881e11a16 1.21 2019-01-16 20:44:38 -08:00
bingoohuang
ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
bingoohuang
6ddfaf33cb extract VolumeFileScanner for ScanVolumeFile 2019-01-16 17:48:59 +08:00
bingoohuang
27093bc2e5 fix needle Append return offset to avoid uint32 overflow 2019-01-16 16:48:03 +08:00
bingoohuang
219838db87
fail fast when failed to write local volume 2019-01-15 15:47:43 +08:00
Chris Lu
2ff95ead57
Merge pull request #832 from PapaYofen/fix-830 
Fix https://github.com/chrislusf/seaweedfs/issues/830
 2019-01-12 17:26:51 -08:00
chenwanli
b71d6f4b88 Fix https://github.com/chrislusf/seaweedfs/issues/830 2019-01-12 10:07:19 +08:00
chenwanli
80cf99db1b Set rpc dial timeout to 5s 2019-01-11 18:23:31 +08:00
Chris Lu
8613e4f558
Merge pull request #827 from suilongfei/master 
change log_dir to logdir for the "flag redefined:log_dir" error when use another package with "github.com/golang/glog"
 2019-01-10 09:10:07 -08:00
Chris Lu
2d23d86fd3 no timeout for volume vacuum 
revert changes on volume vacuum timemout from https://github.com/chrislusf/seaweedfs/pull/829
 2019-01-10 09:07:40 -08:00
chenwanli
39c7455881 Set http timeout to 5s 2019-01-10 20:21:08 +08:00
chenwanli
0a3e83a36a Set timeout for master and volume non-streaming rpc 2019-01-10 19:41:03 +08:00
隋龙飞@视频结构化
b864cf7e07
change log_dir flag to logdir 
change log_dir flag to logdir
 2019-01-10 17:17:32 +08:00
隋龙飞@视频结构化
ea4f29d1e7
change log_dir flag to logdir 
change log_dir flag to logdir
 2019-01-10 17:16:58 +08:00
Chris Lu
2a75a36b27 set cassandra ttl with entry.TTLSec 2019-01-08 20:26:19 -08:00
byungjo.yoon
58fd2e5a19 Set redis key ttl with entry.TTLSec 2019-01-09 11:48:19 +09:00
Chris Lu
badd8fdf8f shortcut for empty file 2019-01-08 09:03:28 -08:00
chenwanli
41fe31b5d0 Speed up heartbeat to new master leader 2019-01-07 16:09:15 +08:00
Chris Lu
e768e64fbc 1.20 2019-01-06 10:52:26 -08:00
Chris Lu
bd32108a90 disable async file deletion 2019-01-05 22:51:50 -08:00
Chris Lu
be9a7592a1 incrementally calculate visible intervals 2019-01-05 22:23:44 -08:00
Chris Lu
6b5d6bb5a6 adjust error names 2019-01-05 19:52:38 -08:00
Chris Lu
9383c91eb1 wait to read again if the volume is compacting 2019-01-05 19:52:17 -08:00
Chris Lu
fe50224ea0 refactoring 2019-01-05 15:21:56 -08:00
Chris Lu
6740427638 weed mount: reduce default chunk size 2019-01-05 15:17:03 -08:00
Chris Lu
d9f9aca453 refactoring 2019-01-05 15:16:39 -08:00
Chris Lu
251b596bc5 increase batch deletion size to 1024 2019-01-03 22:21:57 -08:00
Chris Lu
aac11fb832 reduce directory list cache usage 2019-01-03 22:21:39 -08:00
Chris Lu
7f74577db1 add Last-Modified to HEAD requests 
to work better with restic
 2019-01-02 14:23:25 -08:00
Chris Lu
ceafb7d145 1.19 2019-01-02 12:59:20 -08:00
Chris Lu
28a41fda3e assign new file id even on filer PUT operations 2019-01-02 12:58:26 -08:00
Chris Lu
d67f7ddfaf adjust log 2019-01-02 12:58:06 -08:00
Chris Lu
e49a38842a set etag in response 2019-01-02 12:57:54 -08:00
Chris Lu
d5ebfd5b0c fix s3cmd put 
1. fix volume put data size
 2019-01-02 12:57:33 -08:00
Chris Lu
3339325334 fix s3 ListAllMyBucketsResult to work with s3cmd 2019-01-02 11:36:29 -08:00
Chris Lu
43db7ac123 fix spelling 2019-01-02 02:38:32 -08:00
Chris Lu
9539823c35 fix log 2019-01-01 03:04:44 -08:00
Chris Lu
525db94d9a async file chunk deletion 2019-01-01 02:33:57 -08:00
Chris Lu
ffb5d3f93a get bytes from sync pool 2019-01-01 02:14:40 -08:00
Chris Lu
832ac8129a reduce pointer usage 2018-12-31 15:10:14 -08:00
Chris Lu
1478d7ea21 reduce file seek when writing 2018-12-31 15:08:32 -08:00
Chris Lu
81904ad336 debugging: track all memory allocations 2018-12-31 14:54:13 -08:00
Chris Lu
03cfb4267f adjust vaccum logging 2018-12-31 00:06:52 -08:00
Chris Lu
d6181076bb FUSE remove adjust error 2018-12-30 22:29:27 -08:00
Chris Lu
16e8610426 1.18 2018-12-30 00:52:20 -08:00
Chris Lu
0388d421d2 caching visible intervals for read 
speeds up 4x in single thread mode
speeds up 30% in 32 threads mode
 2018-12-30 00:51:44 -08:00
Chris Lu
55f7d8f801 remove extra log 2018-12-29 18:00:19 -08:00
Chris Lu
170ed53da5 weed mount: mac add AutoXattr 2018-12-29 13:55:38 -08:00
Chris Lu
d8c7844d2e 1.17 2018-12-29 13:16:38 -08:00
Chris Lu
9f3836072a weed mount: no mac finder broswe mode 2018-12-29 13:16:23 -08:00
Chris Lu
d7bf6c5b91 switching_to_seaweedfs_fuse 2018-12-29 10:31:36 -08:00
Chris Lu
0c932d1738 add volume delete grpc API 
fix https://github.com/chrislusf/seaweedfs/issues/808
 2018-12-29 00:03:30 -08:00
Chris Lu
03c7447ad6 set mount point to existing permissions 
fix https://github.com/chrislusf/seaweedfs/issues/806
 2018-12-28 23:36:13 -08:00
Chris Lu
a8af1e3f5f set attributes include crtime 2018-12-28 22:37:18 -08:00
Chris Lu
ba9b7b62d7 1.16 2018-12-28 15:10:49 -08:00
Chris Lu
79168ab0bb adjust error and log 2018-12-28 15:10:23 -08:00
Chris Lu
308ac1d0d2 reuse buffer by sync pool 2018-12-28 03:27:48 -08:00
Chris Lu
260b038441 fix url 2018-12-27 14:43:36 -08:00
Chris Lu
2a19511ba8 avoid gzip benchmark data 
related to https://github.com/chrislusf/seaweedfs/issues/804
 2018-12-27 13:10:46 -08:00
Chris Lu
d3660ffce8 avoid gzipping files without file extensions 
fix https://github.com/chrislusf/seaweedfs/issues/804
 2018-12-27 12:17:05 -08:00
Chris Lu
253f190f48 weed mount add symlink support 2018-12-25 22:45:44 -08:00
Chris Lu
9eb2bee86a avoid error log if not found 2018-12-24 17:45:26 -08:00
Chris Lu
ba923e2864 error if the last arguement does not end with forward slash 2018-12-23 18:20:11 -08:00
Chris Lu
332668a3fd 1.15 2018-12-23 12:54:41 -08:00
Chris Lu
c043fd17cb detect more gzippable content 2018-12-22 15:05:31 -08:00
Chris Lu
49f40cd145 adjust help message 2018-12-22 15:05:15 -08:00
Chris Lu
852ee21835 avoid .gz auto decompression 2018-12-22 13:58:16 -08:00
Chris Lu
be946c9e54 filer store original data size instead of data size after gzip 2018-12-22 13:11:07 -08:00
Chris Lu
141d302492 fix binarySearchCompactSection 2018-12-22 11:52:58 -08:00
Chris Lu
d3839fe279 go fmt 2018-12-22 11:11:57 -08:00
Chris Lu
5333f2984a streaming updates for large entries or large index file 
fix https://github.com/chrislusf/seaweedfs/issues/801
 2018-12-22 11:10:08 -08:00
Chris Lu
36d13355bb compact map fix bug when iterating 2018-12-22 11:05:29 -08:00
Chris Lu
f64dc7465b remove index file 
fix https://github.com/chrislusf/seaweedfs/issues/793
 2018-12-17 20:33:32 -08:00
Chris Lu
16634235c1 1.14 and HCFS 1.0.3 2018-12-17 10:03:03 -08:00
Chris Lu
52712782a2 adding link 2018-12-17 00:31:23 -08:00
Chris Lu
2fb740a2b9 fix upload path 2018-12-17 00:21:34 -08:00
Chris Lu
7b7f4215aa filer update existing entry 2018-12-17 00:20:00 -08:00
Chris Lu
6071be104b adjust log format 2018-12-17 00:03:14 -08:00
Chris Lu
3ac54792e1 paginate when filer deleting and FUSE mount renaming 2018-12-16 23:20:08 -08:00
Chris Lu
93d0ad09af error log format 2018-12-16 17:37:10 -08:00
Chris Lu
66d04cb83f 1.12 2018-12-15 08:17:24 -08:00
Chris Lu
42cb9b76c4 reduce needle map memory usage by 25% 2018-12-15 05:55:56 -08:00
Chris Lu
42b8f1145a weed mount ls paginate the results 
fix https://github.com/chrislusf/seaweedfs/issues/787
 2018-12-11 07:41:50 -08:00
Chris Lu
39ea8484c1 adjust proto API 2018-12-11 07:17:10 -08:00
Chris Lu
168f461761 fix MaxUint8 2018-12-10 20:57:51 -08:00
Chris Lu
8f78a069c6 fix possible name overflow when len(name)>255 2018-12-10 20:57:01 -08:00
Chris Lu
4a490fe35a go fmt 2018-12-09 01:27:25 -08:00
Chris Lu
5741134ca0 change overflow from map to slice 2018-12-09 01:27:11 -08:00
Chris Lu
d9189c857c remove unused file 2018-12-09 00:30:53 -08:00
Chris Lu
df4b9df142 add memory usage report 2018-12-09 00:15:23 -08:00
Chris Lu
ec5a305624 refactor 2018-12-09 00:12:37 -08:00
Chris Lu
9d3be33e5c optimize a little bit 2018-12-08 21:45:14 -08:00
Chris Lu
36d1e739e6 1.11 2018-12-08 00:32:34 -08:00
Chris Lu
1bfb96f34d optimization for reading whole chunk with gzip encoding 2018-12-07 01:57:55 -08:00
Chris Lu
29f1673d97 refactoring 2018-12-07 01:25:01 -08:00
Chris Lu
c28e8a2397 refactoring 2018-12-06 00:44:41 -08:00
Chris Lu
ffa2827ab1 fail fast if two notification queues or inputs are enabled 2018-12-06 00:37:59 -08:00
Chris Lu
63619f8b8e fail early if two filer stores are defined 2018-12-06 00:32:36 -08:00
Chris Lu
52b24a9902 default "weed server -filer" to same directory as -mdir 2018-12-05 23:24:25 -08:00
Chris Lu
fb43a9d7b9 adjust message 2018-12-05 02:03:03 -08:00
chenwanli
31038acccf fix https://github.com/chrislusf/seaweedfs/issues/780 2018-12-05 14:06:16 +08:00
Chris Lu
650d1af978 HCFS support change user name and group names 2018-12-04 00:46:00 -08:00
Chris Lu
b0838db5a8 support hdfs user name and group names 2018-12-04 00:13:40 -08:00
Chris Lu
7ace0efd65 revert prev 2018-12-02 23:25:37 -08:00
Chris Lu
738247d47e do not throw error if not found 2018-12-02 23:20:29 -08:00
Chris Lu
c6b646e92c skip creation for root 2018-12-02 23:16:17 -08:00
Chris Lu
bea162c34c add warning for empty attributes 2018-12-02 22:57:59 -08:00
Chris Lu
f0677c5af1 default root to 0755 2018-12-02 19:59:47 -08:00
Chris Lu
db22666a30 add default root directory 2018-12-02 19:42:50 -08:00
Chris Lu
a3def2bbd9 for 1.10 2018-11-29 09:15:04 -08:00
Chris Lu
7a6f49cd42 refactor a bit more 
fix https://github.com/chrislusf/seaweedfs/issues/777
 2018-11-29 00:07:54 -08:00
Chris Lu
ceca078acb avoid overwriting file or directory 
fix https://github.com/chrislusf/seaweedfs/issues/777
 2018-11-29 00:00:56 -08:00
Chris Lu
1cbd53c01c WIP SeaweedFileSystem added mkdirs, getFileStatus, listStatus, delete 2018-11-25 13:43:26 -08:00
Chris Lu
871dee4674 fix test 2018-11-23 12:37:09 -08:00
Chris Lu
86dd933596 go fmt 2018-11-23 00:26:15 -08:00
Chris Lu
444dfded84 add fs.FSStatfser for SeaweedFS weed mount 2018-11-23 00:24:51 -08:00
Chris Lu
2e32b44061 add template for fs.FSStatfser 2018-11-22 23:05:22 -08:00
Chris Lu
6344469a9b set fuse subtype 2018-11-22 23:04:23 -08:00
Chris Lu
d64546c791 adjust logs 2018-11-22 14:31:10 -08:00
Chris Lu
e12b6c0306 add subimit count parameter 
related to https://github.com/chrislusf/seaweedfs/issues/631
 2018-11-22 13:45:29 -08:00
Chris Lu
0ed816d4e9 formatting error log 2018-11-22 00:42:56 -08:00
Chris Lu
19d546938a forgot to upgrade version number 2018-11-21 23:30:59 -08:00
Chris Lu
60d3681371 default images.fix.orientation to false 2018-11-21 16:50:42 -08:00
Chris Lu
99256100a0 fix tests 2018-11-21 16:35:27 -08:00
Chris Lu
e5ad2223a1 add missing intervals 
fix https://github.com/chrislusf/seaweedfs/issues/774
 2018-11-21 16:25:13 -08:00
Chris Lu
c9b3ef58de options to control filer.export verbosity 2018-11-21 15:15:40 -08:00
Chris Lu
2f2e417b68 detailed export 2018-11-21 15:12:47 -08:00
Chris Lu
b282e34dc2 async file chunk deletion 2018-11-20 20:56:28 -08:00
Chris Lu
5065d4ab2d master add grpc API for fileid assigning 2018-11-20 11:35:45 -08:00
Chris Lu
b318167cd3 batch delete file ids 2018-11-20 11:35:08 -08:00
Chris Lu
06474e5990 1.08 2018-11-18 22:43:05 -08:00
Chris Lu
ee3fe07acf fix sort bug 2018-11-18 21:59:53 -08:00
Chris Lu
469863f7f0 merge interval now is quite faster with reusing slices 2018-11-18 21:24:58 -08:00
Chris Lu
d2924c7748 merge intervals is a bit faster 2018-11-18 20:31:39 -08:00
Chris Lu
cb5a10c6a3 adjust the tests 2018-11-18 17:05:36 -08:00
Chris Lu
f1db22d48b add benchmark test for merging intervals 2018-11-18 16:49:08 -08:00
Chris Lu
4fcfc9410f cleanup 2018-11-18 11:51:38 -08:00
Chris Lu
9655dc9ca9 simpler logic 2018-11-18 10:07:30 -08:00
Chris Lu
f8eb988347 1.07 2018-11-18 08:10:15 -08:00
Chris Lu
60421b23f0 reduce weed mount memory usage 2018-11-18 07:49:14 -08:00
Chris Lu
2929563294 1.06 2018-11-14 23:51:15 -08:00
Chris Lu
d3b7965c76 caching grpc clients 2018-11-14 23:50:46 -08:00
Chris Lu
d89774cb7f cache grpc client connections to improve performance 2018-11-14 23:31:39 -08:00
Chris Lu
2e6a3c7b16 reduce memory usage 2018-11-14 22:48:54 -08:00
Chris Lu
f224b9fe28 1.05 2018-11-13 23:54:17 -08:00
Chris Lu
11b0a9cb76 fix possible error when reusing file handles 2018-11-13 23:53:17 -08:00
Chris Lu
a62adf62ca a bit faster 2018-11-12 11:19:13 -08:00
Chris Lu
affe708961 a bit faster 2018-11-11 00:43:30 -08:00
Chris Lu
a09eaf8756 adjusting, seems there are some perf benefit 2018-11-11 00:08:22 -08:00
Chris Lu
8178fc9973 adjust logging 2018-11-11 00:07:46 -08:00
Chris Lu
2697d6a4fb fix deleting bucket 2018-11-08 22:15:21 -08:00
Chris Lu
0d095f72e4 update to 1.04 2018-11-08 21:42:45 -08:00
Chris Lu
a4ceb051a7 use MD5 for ETag to be consistent with Amazon S3 2018-11-08 21:41:02 -08:00
Chris Lu
6e53c38c2f update to 1.03 2018-11-08 07:37:44 -08:00
Chris Lu
cbd94b18a5 improve "ls -al" performance for large directory 2018-11-08 07:37:34 -08:00
Chris Lu
6e11923551 upgrade to 1.02 2018-11-07 11:37:32 -08:00
Chris Lu
ed8efb5aef much improved "ls -al" performance 2018-11-07 11:35:13 -08:00
Chris Lu
60db731e36 update to 1.01 2018-11-05 08:54:40 -08:00
Chris Lu
f9dbaf6292 delete file by name instead of referencing the dataFile pointer 2018-11-05 08:53:38 -08:00
alex8224
8d3f9c0b75 fix invalid point when delete collection 2018-11-05 19:53:55 +08:00
Chris Lu
1a64a817ac update to 1.00 2018-11-04 22:13:16 -08:00
Chris Lu
85d1d99459 default "weed export to current enabled store" 2018-11-04 12:59:29 -08:00
Chris Lu
6c45787a37 selectively export files 2018-11-04 12:07:33 -08:00
Chris Lu
ff4189ab25 go fmt 2018-11-04 11:59:08 -08:00
Chris Lu
8dfac6a4cf working b2 sink 2018-11-04 11:58:59 -08:00
Chris Lu
f050b22d6c close file in ScanVolumeFile() 
fix https://github.com/chrislusf/seaweedfs/issues/761
 2018-11-04 00:28:24 -07:00
Chris Lu
93e0dddbf3 add function description 2018-11-03 22:32:21 -07:00
Chris Lu
a64613172d bootstrap filer replication with weed filer.export -targetStore=notification 2018-11-03 12:43:45 -07:00
Chris Lu
3674ad9f8e go fmt 2018-11-01 01:12:21 -07:00
Chris Lu
db584ff7f8 separate into notification.toml, add gcp pub/sub message queue 2018-11-01 01:11:09 -07:00
Chris Lu
6219a9ad1f filer ordered notifications 2018-10-31 21:48:05 -07:00
Chris Lu
4c97ff3717 support AWS SQS as file change notification message queue 2018-10-31 01:11:19 -07:00
Chris Lu
200cbcde62
Merge pull request #756 from zzoe/patch-1 
Update volume_server_handlers_write.go
 2018-10-30 21:52:19 -07:00
zoe
a11c28cb4c
Update volume_server_handlers_write.go 
size declared and not used
 2018-10-31 09:36:32 +08:00
Chris Lu
fe31f389b0 add configuration instructions 2018-10-30 02:29:11 -07:00
zoe
29eff0ecd7
Update volume_server_handlers_write.go 
返回实际文件大小，而不是needle的大小
 2018-10-29 21:05:51 +08:00
Chris Lu
9e98bba206 go fmt 2018-10-24 00:00:01 -07:00
Chris Lu
6a756136ef go fmt 2018-10-23 23:59:49 -07:00
Chris Lu
c58b73ad9b adding BackBlaze, Azure 2018-10-23 23:59:40 -07:00
Chris Lu
b861899919 use released version of azblob 
replace https://github.com/chrislusf/seaweedfs/pull/754
 2018-10-23 02:18:46 -07:00
Chris Lu
82a5d39778 add back volume vacuum cleanup 2018-10-18 23:02:58 -07:00
Chris Lu
496a7398d1 adjust logging 2018-10-18 20:47:30 -07:00
Chris Lu
201c24c110 add locking to volume layout vacuum 2018-10-18 20:34:43 -07:00
Chris Lu
46eb77f9bb move DiskStatus and MemStatus to protobuf 2018-10-15 22:25:28 -07:00
Chris Lu
eec951cad2 migrate volume sync to gRpc 2018-10-15 21:44:41 -07:00
Chris Lu
f8b2d3cacc move volume mount/unmount on volume server to grpc 2018-10-15 01:48:15 -07:00
Chris Lu
66a353dcb5 remove volume server /admin/volume/delete 2018-10-15 01:26:49 -07:00
Chris Lu
fda771c83f migrate volume sync status to grpc API on volume server 2018-10-15 01:19:15 -07:00
Chris Lu
b423bb9e2d migrate assign volume to grpc API on volume server 2018-10-15 00:40:46 -07:00
Chris Lu
333709657c adjust log level 2018-10-15 00:40:12 -07:00
Chris Lu
f9410bcaa7 adjust log level for deletes 2018-10-15 00:39:56 -07:00
Chris Lu
8301519fb0 migrate delete collection to grpc API on volume server 2018-10-15 00:03:55 -07:00
Chris Lu
db152ca540 fix error log format 2018-10-14 23:50:54 -07:00
Chris Lu
67d1e572ca rename files 2018-10-14 23:47:29 -07:00
Chris Lu
b1daede91b move volume vacuum to gRpc 2018-10-14 23:12:43 -07:00
Chris Lu
91ac2e0dd9 go fmt 2018-10-14 00:30:20 -07:00
Chris Lu
249eeccf32 0.99 release 2018-10-14 00:19:04 -07:00
Chris Lu
9db53cf256 use the configurable chunk size 2018-10-14 00:18:52 -07:00
Chris Lu
ff66269b62 use grpc to replace http APIs for batch volume id lookup and batch delete 
1. remove batch volume id lookup http API /vol/lookup
2. remove batch delete http API /delete
 2018-10-14 00:12:28 -07:00
Chris Lu
3ddcd87098 increase weed mount IO perforance 2018-10-14 00:10:17 -07:00
Chris Lu
ab85118233 error handling when kafka not ready 2018-10-13 23:30:00 -07:00
Chris Lu
41aba0c02c fix when no files in filer 2018-10-12 00:45:28 -07:00
Chris Lu
4af6b4eb0e fix master static resource display problem 2018-10-12 00:05:15 -07:00
Chris Lu
556382ff5f adding basic grpc to volume server 2018-10-11 01:16:33 -07:00
Chris Lu
64ed47502b rename file 2018-10-11 00:21:32 -07:00
Chris Lu
08266b7256 go fmt 2018-10-11 00:08:13 -07:00
Chris Lu
1d7e01b754 reformat 2018-10-11 00:05:54 -07:00
Chris Lu
da6154b29c refactor volume server to startVolumeServer() 2018-10-11 00:04:31 -07:00
Chris Lu
60d2f1557d refactor func name 2018-10-10 23:19:54 -07:00
Chris Lu
b7763c1cb6 releasing 0.98 2018-10-10 21:44:10 -07:00
Chris Lu
9f4c2f87d3 adding filer replication sink to Azure 2018-10-09 01:35:48 -07:00
Chris Lu
453d0be4d4 adjust error format 2018-10-09 01:35:30 -07:00
Alex Wang
4db68669b6 [bugfix] Fix interrupt hook overwritten bug 2018-10-09 15:43:32 +08:00
Chris Lu
96a23591e7 remove comments 2018-10-07 22:21:00 -07:00
Chris Lu
dbe33236fb embed static resources via statik 2018-10-07 21:42:57 -07:00
Chris Lu
fffbc2d490 embed static resources via statik 2018-10-07 10:54:05 -07:00
Chris Lu
a5b5887efc add filer replication to aws s3 2018-10-06 17:10:15 -07:00
Chris Lu
84d4a2c8a0 add filer replication to google cloud storage 2018-10-06 13:08:38 -07:00
Chris Lu
f8c2704d2b reformat 2018-10-06 13:04:33 -07:00
Chris Lu
14858f6224 add logging 2018-10-06 13:01:38 -07:00
Chris Lu
04da4c8094 add logging 2018-10-06 13:01:29 -07:00
Chris Lu
4a8ef198d7 add logging 2018-10-06 13:00:33 -07:00
Chris Lu
56c5c7b1b6 add google cloud storage 2018-10-04 01:14:44 -07:00
Chris Lu
e8ef501f02 add s3 replication sink 2018-10-03 23:36:52 -07:00
Chris Lu
56a5d5af8d ignore duplicated key error if the store has duplicated primary key prevention 
fix https://github.com/chrislusf/seaweedfs/issues/741
 2018-09-30 00:49:52 -07:00
Chris Lu
98384b62fd adjust description 2018-09-28 01:58:34 -07:00
Chris Lu
f11d531d45 adjust logging 2018-09-28 01:54:10 -07:00
Chris Lu
06c43286cd skip error when deletion failed 2018-09-28 00:10:49 -07:00
Chris Lu
746017ea4f releasing 0.97 2018-09-27 21:51:41 -07:00
Chris Lu
f7ff84f17f Revert "smalller block size for better accuracy" 
This reverts commit 97ecd20fc9.
 2018-09-27 09:42:24 -07:00
Chris Lu
97ecd20fc9 smalller block size for better accuracy 2018-09-27 09:39:17 -07:00
Chris Lu
f53632a84b add block size for linux du 
du already working on Mac
 2018-09-27 09:33:36 -07:00
Chris Lu
a362261972 avoid possible racing issue if leveldb or boltdb is used. 
remove the to-be-generated files during commit

fix https://github.com/chrislusf/seaweedfs/issues/738
 2018-09-26 18:45:51 -07:00
Chris Lu
31ed352ab6 replication handle cases when entry already exists 2018-09-25 09:27:03 -07:00
Chris Lu
60c1ada4c1 remove filer grpc API GetEntryAttributes 2018-09-25 09:18:06 -07:00
Chris Lu
578f8a6dec fix empty file reading on linux 2018-09-23 10:01:00 -07:00
Chris Lu
200d9b9ac2 add file name to log 2018-09-23 09:48:55 -07:00
Chris Lu
a60767c43a add more instructions 2018-09-23 09:26:39 -07:00
Chris Lu
12ba8b9799 generate useable filer.toml 2018-09-23 09:26:25 -07:00
Chris Lu
b1b8c4ed32 join via filepath 2018-09-23 01:46:50 -07:00
Chris Lu
b16d47030e save and use last processed kafka offsets 2018-09-23 01:34:40 -07:00
Chris Lu
9fe24991d5 refactoring 2018-09-23 00:40:36 -07:00
Chris Lu
7d6b2a4740 add ETag to upload results and chunks 2018-09-22 22:12:21 -07:00
Chris Lu
420f068376 add more instructions 2018-09-22 22:12:06 -07:00
Chris Lu
d3e1f20dcc adjust log level 2018-09-22 22:11:49 -07:00
Chris Lu
13d6a74eb2 consolidate file chunks during create 2018-09-22 22:11:37 -07:00
Chris Lu
fdc8a267de remove unused function 2018-09-22 22:11:13 -07:00
Chris Lu
9a3d46f8e3 handle duplicated replication when retry happens 2018-09-22 11:14:04 -07:00
Chris Lu
01ceace18e adjust sink options 2018-09-22 00:53:52 -07:00
Chris Lu
402ab598b6 go fmt 2018-09-22 00:12:10 -07:00
Chris Lu
7bb62b9f21 delay file creation without chunks from dir.Create to file.Flush 2018-09-22 00:11:46 -07:00
Chris Lu
db69ce89f0 go fmt 2018-09-21 01:56:43 -07:00
Chris Lu
a6cfaba018 able to sync the changes 2018-09-21 01:54:29 -07:00
Chris Lu
a8cd7df44a let filer choose default replication 2018-09-20 09:00:15 -07:00
Chris Lu
d5149e592e add logging on kafka reading 2018-09-19 22:31:45 -07:00
Chris Lu
d9b32db8d7 adjust uploads to a hidden folder 2018-09-19 22:03:16 -07:00
Chris Lu
f5471bcebf add bucket name in the redirection 2018-09-19 22:01:41 -07:00
Chris Lu
25fb6f9a46 fix compilation 2018-09-17 02:23:21 -07:00
Chris Lu
779641e9d4 adjust replicated entry name 2018-09-17 01:37:24 -07:00
Chris Lu
b7a375f5aa set large limit for "weed filer.export" directory listing 2018-09-17 00:42:36 -07:00
Chris Lu
4e1f1a0b92 set large limit for "weed mount" directory listing 2018-09-17 00:41:33 -07:00
Chris Lu
788acdf527 add WIP filer.replicate 2018-09-17 00:27:56 -07:00
Chris Lu
865a017936 fix when if buffer is not aligned 2018-09-17 00:27:40 -07:00
Chris Lu
97575e7185 do not set attributes if the file is still open 2018-09-16 12:37:06 -07:00
Chris Lu
70e67b7c6d add notification log 2018-09-16 11:20:36 -07:00
Chris Lu
5c6e3f457f adjust log 2018-09-16 11:20:08 -07:00
Chris Lu
bc03233364 refactoring 2018-09-16 01:37:35 -07:00
Chris Lu
d923ba2206 renaming msgqueue to notification 2018-09-16 01:18:30 -07:00
Chris Lu
bea4f6ca14 go fmt 2018-09-12 01:00:57 -07:00
Chris Lu
f628953c33 cleanup tmp upload directories 2018-09-12 01:00:51 -07:00
Chris Lu
f6d8525d1d working S3 multipart uploads 2018-09-12 00:46:12 -07:00
Chris Lu
439e88b5a6 fix format error 2018-09-11 13:01:51 -07:00
Chris Lu
267201ff44 handle non continuous writes 
fix #728
 2018-09-10 02:39:41 -07:00
Chris Lu
109385e955 simplifying logic to avoid handling non continuous writes 
fix https://github.com/chrislusf/seaweedfs/issues/728
 2018-09-10 02:21:57 -07:00
Chris Lu
296acc4a0a go fmt 2018-09-09 16:26:11 -07:00
Chris Lu
164091c269 add s3 multipart upload 2018-09-09 16:25:43 -07:00
Chris Lu
9b3bf0e46c fix "weed backup" rerunning 
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
 2018-09-09 02:48:58 -07:00
Chris Lu
bc8d34143d Revert "fix "weed backup" rerunning" 
This reverts commit 1937be2bb2.
 2018-09-09 02:38:30 -07:00
Chris Lu
1937be2bb2 fix "weed backup" rerunning 
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
 2018-09-09 02:31:12 -07:00
Chris Lu
d7a4b12d2a fix build error 2018-09-07 14:21:10 -07:00
Chris Lu
83ca595472 adjust version number 2018-09-07 14:19:48 -07:00
Chris Lu
747377142b fix file permission during writes 2018-09-07 13:57:06 -07:00
Chris Lu
dd37f0540f add extended attributes 2018-09-07 13:12:52 -07:00
Chris Lu
75e749039b fix data writes error when consecutive calls overlaps with previous writes 2018-09-07 13:11:43 -07:00
Chris Lu
f177cec509 avoid slice out of bounds 
avoid this problem

2018/09/04 16:27:14 fuse: panic in handler for Write [ID=0x27c0d Node=0x2 Uid=0 Gid=0 Pid=0] 0x1 131072 @10607788032 fl=WriteCache lock=0 ffl=OpenReadOnly: runtime error: slice bounds out of range
goroutine 211141 [running]:
bazil.org/fuse/fs.(*Server).serve.func2(0x10d3e60, 0xc00014be30, 0xc00052fef8, 0xc00052fe77)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:857 +0x1ac
panic(0xe2d080, 0x17f62b0)
	/home/travis/.gimme/versions/go/src/runtime/panic.go:513 +0x1b9
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).saveToStorage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0xc0005dc000, 0x20000, 0x1000000, 0x276720000, 0xc0003feaa0, 0x0, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:142 +0x8ec
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).saveExistingPagesToStorage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0x0, 0x0, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:107 +0x6c
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).AddPage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0x278460000, 0xc011966050, 0x20000, 0x20fb0, 0x6fc23ac00, 0x4a817c800, 0x0, ...)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:70 +0x8f
github.com/chrislusf/seaweedfs/weed/filesys.(*FileHandle).Write(0xc000548410, 0x10d7ba0, 0xc0003fcc00, 0xc00014be30, 0xc011946af8, 0x47fa01, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/filehandle.go:141 +0x245
bazil.org/fuse/fs.(*Server).handleRequest(0xc0002cc0c0, 0x10d7ba0, 0xc0003fcc00, 0x10cb020, 0xc000394140, 0xc0000acac0, 0x10d3e60, 0xc00014be30, 0xc00052fef8, 0x10ca6a0, ...)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:1265 +0x1599
bazil.org/fuse/fs.(*Server).serve(0xc0002cc0c0, 0x10d3e60, 0xc00014be30)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:878 +0x410
bazil.org/fuse/fs.(*Server).Serve.func1(0xc0002cc0c0, 0x10d3e60, 0xc00014be30)
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:425 +0x6e
created by bazil.org/fuse/fs.(*Server).Serve
	/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:423 +0x321
 2018-09-05 02:17:04 -07:00
Chris Lu
d614716ff2 fix compilation error 2018-09-04 00:43:46 -07:00
Chris Lu
ce6a2140a2 preparing to support S3 multipart uploads 2018-09-04 00:42:44 -07:00
Chris Lu
7f760f16b0 prepare for 0.95 2018-09-03 20:52:52 -07:00
Chris Lu
35bafd9861 remove unused function 2018-09-03 16:50:32 -07:00
Chris Lu
7ad5a90127 go fmt 2018-09-03 16:47:00 -07:00
Chris Lu
2ed6d8cca6 refactoring filer rm 2018-09-03 13:16:26 -07:00
Chris Lu
03f852c799 refactoring 2018-09-03 13:03:16 -07:00
Chris Lu
0b0ece9649 add place holders for multiplarts upload 2018-09-03 11:38:10 -07:00
Chris Lu
0adb2ba6b8 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2018-09-03 00:29:02 -07:00
Chris Lu
06b81c063a fix possible connection counting error 2018-09-03 00:28:58 -07:00
zhangmingfeng
aafe8d84cb
Update volume_server_handlers_read.go 
change ext to lower-case
 2018-09-03 11:01:35 +08:00
Chris Lu
88f1d32cc4 add s3ChunkedReader 
fix https://github.com/chrislusf/seaweedfs/issues/718
 2018-09-02 14:20:47 -07:00
Chris Lu
76cbe8bf33 instant notification of new volumes added or deleted 2018-08-24 01:26:56 -07:00
Chris Lu
5ccf8e8078 reformat 2018-08-24 00:30:03 -07:00
Chris Lu
ac793a3c5a clean remove all left over files when deleting a collection 2018-08-23 23:33:16 -07:00
Chris Lu
d361a1ddf5 fix id String() instead 2018-08-23 22:44:30 -07:00