mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-23 17:07:57 +08:00
Code Issues Packages Projects Releases Wiki Activity
5,566 Commits 32 Branches 296 Tags 152 MiB
d9a2a7f1c4
Commit Graph

304 Commits

Author SHA1 Message Date
Chris Lu
da7e5aaa65 filer: optimize for less number of directory lookup 
bottom up directory lookup
 2020-12-27 20:46:17 -08:00
Chris Lu
da87f6b265 remove unused code 2020-12-26 15:21:12 -08:00
Chris Lu
0a7c5f85a9 filer: add namePattern to search in current folder 2020-12-26 15:05:31 -08:00
Chris Lu
0e016bc7bd hbase add ttl 2020-12-24 12:10:35 -08:00
Chris Lu
9bf6c10505 fix prefix 2020-12-24 00:18:59 -08:00
Chris Lu
2fea8cfa0f fix compilation 2020-12-23 23:54:15 -08:00
Chris Lu
53bc1ea25b fix compilation 2020-12-23 23:53:46 -08:00
Chris Lu
b5e2be635a adjust for directory listing 2020-12-23 23:49:22 -08:00
Chris Lu
c4a202ec41 fix wrong column family 2020-12-23 23:23:05 -08:00
Chris Lu
94e3757c08 fix 2020-12-23 21:45:16 -08:00
Chris Lu
64c48c9724 seems compiling 2020-12-23 21:44:53 -08:00
Chris Lu
1c7e1295dc Merge branch 'master' into support_ssd_volume 2020-12-22 17:44:52 -08:00
Chris Lu
a0990b929d minor 2020-12-22 17:33:40 -08:00
Chris Lu
97e3432dfe avoid wrong error 
fix https://github.com/chrislusf/seaweedfs/issues/1691
 2020-12-22 02:46:24 -08:00
Chris Lu
4f31c1bb94 go fmt 2020-12-22 02:34:08 -08:00
Chris Lu
6c4f32d173 remove unused code 2020-12-22 02:33:31 -08:00
Chris Lu
90d785a15f filer: redis, redis cluster, cassandra support super large directory 2020-12-22 02:26:05 -08:00
Chris Lu
fe46411cd4 handle directory listing edge cases 2020-12-21 23:37:43 -08:00
Chris Lu
c74bede730 handle listing directories 2020-12-21 23:19:05 -08:00
Chris Lu
adf8cb4000 filer: path-specific stores trim out common prefixes 2020-12-21 22:57:13 -08:00
Chris Lu
0823bde8d8 refactor: split into two files 2020-12-21 22:05:15 -08:00
Chris Lu
8ab70ccf76 refactor 2020-12-21 21:46:32 -08:00
Chris Lu
65cc6dc636 refactor to dedicated function 2020-12-21 21:18:34 -08:00
Chris Lu
488c2680e8 change default filer store root 2020-12-21 21:15:22 -08:00
Chris Lu
f5a19a87be filer: report error if the path specific store is not found 
related https://github.com/chrislusf/seaweedfs/issues/1691
 2020-12-21 12:43:52 -08:00
Chris Lu
41c0f3ad24 filer: support path-specific filer store 2020-12-19 01:27:09 -08:00
Chris Lu
0d5683fb0e todo: load path-specific store from filer.toml 2020-12-18 03:05:42 -08:00
Chris Lu
3269fd7eaf filer: use store by path 2020-12-18 02:57:49 -08:00
Chris Lu
e605f1e001 refactor, add getActualStore() by path 2020-12-18 02:55:00 -08:00
Chris Lu
23903aa95b refactoring 2020-12-18 02:35:45 -08:00
Chris Lu
4e6b316913 assert FilerStoreWrapper is VirtualFilerStore 2020-12-18 02:17:06 -08:00
Chris Lu
1bf22c0b5b go fmt 2020-12-16 09:14:05 -08:00
Chris Lu
c45d6a865b Merge branch 'master' into support_ssd_volume 2020-12-13 22:14:18 -08:00
Chris Lu
38fc89d041 filer store: sql update if any insert error happens 
fix https://github.com/chrislusf/seaweedfs/issues/1673
 2020-12-13 20:49:44 -08:00
Chris Lu
23280257df sql put kv: avoid unnecessary update 2020-12-13 19:45:47 -08:00
Chris Lu
0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu
e9cd798bd3 adding volume type 2020-12-13 00:58:58 -08:00
Chris Lu
14910d035c the pagination size was too big for recursive deletion 2020-12-12 13:26:10 -08:00
Chris Lu
03637d6f57 s3: move "delete-directory-if-empty" to read time 
move "delete-directory-if-empty" to read time instead of entry deletion time

the listing speed for a s3 bucket folder will slow down if it has many sub folders

related to 0d345ac97d

fix https://github.com/chrislusf/seaweedfs/issues/1647

fix https://github.com/chrislusf/seaweedfs/issues/1670
 2020-12-12 03:38:34 -08:00
Chris Lu
37075a414d adjust logs 2020-12-12 03:33:57 -08:00
Chris Lu
3fedfec1e7 check cross device rename error 2020-12-10 23:50:32 -08:00
Chris Lu
c2f18a10cb minor 2020-12-10 22:23:22 -08:00
Chris Lu
4b0c2a846b add logs 2020-12-10 19:55:28 -08:00
Chris Lu
83078ac6ce filer: change to /etc/seaweedfs folder on filer 
fix https://github.com/chrislusf/seaweedfs/issues/1666
 2020-12-10 11:11:02 -08:00
Chris Lu
765b3ef1be save /etc/iam/identity.json inside filer store 2020-12-10 00:15:22 -08:00
Chris Lu
a9c619c270 added error handling for saving 2020-12-09 01:34:24 -08:00
Chris Lu
8e78187a97 add back last read chunk cache to reader and properly close the reader 2020-12-08 22:26:46 -08:00
Chris Lu
900d22c6ec mount: avoid memory leaking read buffer 
fix https://github.com/chrislusf/seaweedfs/issues/1654

the reader goes together with the file handle, which may stay for a long time.
 2020-12-08 02:38:53 -08:00
Chris Lu
fce8803087 break import cycle 2020-12-07 00:29:17 -08:00
Chris Lu
01e2da5782 refactoring 2020-12-06 23:16:20 -08:00
Chris Lu
9acda432fe fix import cycle 2020-12-06 20:12:52 -08:00
Chris Lu
5931a2f53f filer conf: support json conf only 2020-12-06 19:47:06 -08:00
Chris Lu
f4abd01adf filer: cache small file to filer store 2020-11-30 04:34:04 -08:00
Chris Lu
a9c6be5fc3 set creation time 2020-11-30 03:11:52 -08:00
Chris Lu
0d345ac97d s3: remove empty parent folder on delete 
fix https://github.com/chrislusf/seaweedfs/issues/1637
 2020-11-26 12:21:58 -08:00
Chris Lu
cc2bd97ad9 refactor 2020-11-26 11:25:56 -08:00
Chris Lu
0da7ecfd29 go fmt 2020-11-26 11:22:30 -08:00
Chris Lu
effa00ed08 refactor 2020-11-26 11:21:58 -08:00
Chris Lu
513bcd6e0d filer: avoid duplicated FindEntry for deletion 2020-11-26 11:14:56 -08:00
Chris Lu
eab53ea80d filer leveldb store: a bit more efficient directory listing with prefix 2020-11-22 21:10:41 -08:00
Chris Lu
290b5e2cd0 directly delete file chunks 
keeping current async deletions for now
 2020-11-17 17:20:21 -08:00
Chris Lu
dc304342b2 fs.configure: configurable volume growth 2020-11-17 01:00:02 -08:00
Chris Lu
5f19e81dab filer confi: support hierachical configuration 2020-11-16 16:50:12 -08:00
Chris Lu
c0d279c54e filere.conf: prefer to use json format 2020-11-15 21:48:17 -08:00
Chris Lu
ee2fa14dbe filer conf: delete location specific configuration 2020-11-15 20:15:47 -08:00
Chris Lu
71056dae07 fs.configure: read and local add filer configuration 2020-11-15 18:09:35 -08:00
Chris Lu
0ea5c087ce go fmt 2020-11-15 16:59:28 -08:00
Chris Lu
95c0de285d refactoring 2020-11-15 16:58:48 -08:00
Chris Lu
500bcab953 refactoring 2020-11-15 14:41:56 -08:00
Chris Lu
590f02179d filer: load filer conf when starting 2020-11-15 14:06:03 -08:00
Chris Lu
0a406f652e load filer conf and match by prefix 2020-11-15 00:26:05 -08:00
Chris Lu
0fc9ffc603 able to read filer.conf if updated 2020-11-14 21:21:20 -08:00
Chris Lu
442e092995 filer watch "/etc" folder for configuration changes 2020-11-14 14:26:08 -08:00
Konstantin Lebedev
aa30604cec fix tests 2020-11-12 08:49:38 +05:00
Konstantin Lebedev
1eec5c8d5d gen pb 2020-11-12 04:10:06 +05:00
Konstantin Lebedev
fc7baef5bb fiil serverUrls sorted by data center 2020-11-12 02:13:33 +05:00
Chris Lu
8750cac090 move to util.RetryWaitTime 2020-11-01 02:36:43 -08:00
Chris Lu
df8d976bb0 refactoring 2020-11-01 01:58:48 -07:00
Chris Lu
16fae84414 fix format error 2020-10-24 20:11:31 -07:00
Chris Lu
e9d40b80b2 less verbose logs 2020-10-24 09:42:54 -07:00
Chris Lu
b81359823f postgres: support empty user 2020-10-22 14:27:47 -07:00
Chris Lu
d75bc62196 to read files that are quickly changing 2020-10-22 12:59:15 -07:00
Chris Lu
5179e559f7 skip empty logs 2020-10-22 00:35:48 -07:00
Chris Lu
bd103c143a add lock for vidCache 2020-10-21 19:28:59 -07:00
Chris Lu
68d39c86f1 mysql, postgres, cassandra: change kv key to base64 encoding 
The exisitng key-value operation for stores using mysql, postgres, and maybe cassandra are already broken.

The kv is used to store hardlink, filer store signature and replication progress.

So users using hardlink and also uses mysql, postgres, or cassandra will have broken hard links.

Users using filer.sync will need to re-sync the files.
 2020-10-16 11:10:12 -07:00
Chris Lu
c95d3d9dff adjust tests 2020-10-14 12:27:52 -07:00
Chris Lu
93bcf56514 file read report EOF 
fix https://github.com/chrislusf/seaweedfs/issues/1344
 2020-10-14 12:18:24 -07:00
Chris Lu
0542911e29 go fmt 2020-10-13 19:50:46 -07:00
Chris Lu
58fa506491 minor 2020-10-13 19:50:22 -07:00
Chris Lu
aac4cb1f0c adds errror on read and write 2020-10-13 13:53:34 -07:00
Chris Lu
3f7d1d1bf1 Only wait on retryable requests 2020-10-13 00:29:46 -07:00
Chris Lu
723ae11db4 refactoring in order to adjust volume server url later 2020-10-11 20:15:10 -07:00
Chris Lu
d155f907c2 mount: configurable read wait time 2020-10-10 20:09:43 -07:00
Chris Lu
8a52379ecb add retry if volume can not be found 2020-10-10 16:02:39 -07:00
Chris Lu
70af0ec24c filer: fix hanging on read 2020-10-10 16:02:10 -07:00
Chris Lu
cff8bb6554 return proper error 2020-10-10 15:43:22 -07:00
Chris Lu
6da8eef54f fix logic error 2020-10-09 00:01:47 -07:00
Chris Lu
5d01dd28a6 add logging 2020-10-08 23:38:00 -07:00
Chris Lu
8d34eb0050 mount:exponentially backoff if read error for about 10 minutes 2020-10-08 23:31:26 -07:00
Chris Lu
6e1f936efd refactoring 2020-10-08 23:19:42 -07:00
Chris Lu
b2ee5873fb fix error not being returned 2020-10-08 23:19:20 -07:00
Chris Lu
8676db0f6a adjust logging 2020-10-08 18:33:37 -07:00
Chris Lu
eed492b73b randomize file locations 2020-10-07 23:58:32 -07:00
Chris Lu
62d3d3aea0 go fmt 2020-10-07 23:30:54 -07:00
Chris Lu
a8624c2e4f read from alternative replica 
related to https://github.com/chrislusf/seaweedfs/issues/1512
 2020-10-07 22:49:04 -07:00
Chris Lu
36492c47ec adjust 2020-10-05 14:06:18 -07:00
Chris Lu
8f8e9ddbde adjust API 2020-10-05 09:47:07 -07:00
Konstantin Lebedev
e4f2d9eb4a We return etag using the same algorithm as aws s3 
https://teppen.io/2018/06/23/aws_s3_etags/
 2020-10-05 14:43:32 +05:00
Chris Lu
6b591b02af adjust log level 2020-10-04 16:21:43 -07:00
Chris Lu
666859f7cd mount: read prefetching 2020-10-04 01:31:04 -07:00
Chris Lu
bc47835997 mount: avoid duplicated reads 2020-10-03 20:16:42 -07:00
Chris Lu
8d65ad1444 Revert "mount: adds read prefetching" 
This reverts commit 692f0614d1.
 2020-10-03 19:40:56 -07:00
Chris Lu
692f0614d1 mount: adds read prefetching 2020-10-03 16:50:09 -07:00
limd
ed7816681a Add cassandra authenticator mode 2020-09-25 11:22:47 +08:00
Chris Lu
0790c6d605 fix empty hard link id 2020-09-24 11:16:43 -07:00
Chris Lu
1012df7bb5 switch hardlink id from int64 to bytes 2020-09-24 11:11:42 -07:00
Chris Lu
2e7c361a0d hardlink deletion factors in hardlink counter 2020-09-24 09:43:52 -07:00
Chris Lu
1295347958 adjust hardlink update 
simplify logic, pass entity content directly to hard link. The "weed mount" handles the logic to calculate hard link counter.
 2020-09-24 09:43:00 -07:00
Chris Lu
5e239afdfc hardlink works now 2020-09-24 03:06:48 -07:00
Chris Lu
b61d33f251 expose only store wrapper to meta changes 2020-09-22 16:27:36 -07:00
Chris Lu
852e5f7cbc filer: fix mongodb insert 
fix https://github.com/chrislusf/seaweedfs/issues/1471
 2020-09-17 21:50:52 -07:00
Chris Lu
10f9081526 filer: adjust meta data events to received timestamp 
if a client is already connected and start from t0. A message recieved at t+1 but with timestamp t-1 may not be processed by the client.

This commit changes to the event received time, so the replication can be ordered.
 2020-09-15 01:18:33 -07:00
Chris Lu
b0c7de186d filer: fix postgres prefixed directory listing problem 
fix https://github.com/chrislusf/seaweedfs/issues/1465
 2020-09-12 13:37:03 -07:00
Chris Lu
446e476a11 go fmt 2020-09-12 04:08:03 -07:00
Chris Lu
2a0925590c filer: etcd store avoid read with nil option 
fix https://github.com/chrislusf/seaweedfs/issues/1463
 2020-09-11 15:29:45 -07:00
ruitao.liu
5b0676049a change elastic initialize process similar as others. 2020-09-10 23:35:20 +08:00
ruitao.liu
9be4e97625 change logs print format. 2020-09-10 16:30:15 +08:00
ruitao.liu
719dc43af1 modify elastic urls from string to array. 2020-09-10 16:24:09 +08:00
ruitao.liu
72f9d7f047 use util to generate md5. 2020-09-10 16:11:18 +08:00
ruitao.liu
6a5b38c0d4 fix elastic kv ops. 2020-09-10 15:59:16 +08:00
ruitao.liu
3f7fbfddca add more basic elastic options. 2020-09-10 14:22:07 +08:00
Chris Lu
daf0a449f7 properly cancel context for streaming grpc 2020-09-09 12:07:15 -07:00
Chris Lu
387ab6796f filer: cross cluster synchronization 2020-09-09 11:21:23 -07:00
Chris Lu
44b3f2efc8 filer: Elastic Search return correct kv error 
fix https://github.com/chrislusf/seaweedfs/issues/1452
 2020-09-07 12:44:51 -07:00
Chris Lu
d8af6f62df adjust logging 2020-09-06 12:10:40 -07:00
Chris Lu
07f32feed0 print sync progress 2020-09-06 00:50:51 -07:00
Chris Lu
0af6252b2d fix compilation 2020-09-06 00:50:38 -07:00
Chris Lu
ac494ff5e8 Update meta_aggregator.go 2020-09-06 00:29:16 -07:00
Chris Lu
c9f8f25ba5 read peer filer from start 2020-09-06 00:12:41 -07:00
Chris Lu
30dc365cbd add self filer 2020-09-06 00:11:46 -07:00
Chris Lu
8acd714600 apply meta changes only if store is different 2020-09-05 23:25:03 -07:00
Chris Lu
d741ed66db check whether shares the same filer store 2020-09-05 23:02:52 -07:00
Chris Lu
bba90ff3c8 read filer signature 2020-09-05 22:52:15 -07:00
Chris Lu
71b0e256c7 filer: setOrLoadFilerStoreSignature 2020-09-05 14:08:59 -07:00
ruitao.liu
450cf07505 skip the index that for kv usage. 2020-09-04 21:49:03 +08:00
ruitao.liu
1384ff9a2f 1.split kv in one file. 
2.disable query for kv in es index.
 2020-09-04 17:34:26 +08:00
ruitao.liu
83080b5e03 ES backended filer support kv ops. 2020-09-04 15:40:13 +08:00
Chris Lu
46f65a84a6 filer: elastic7 adjust default value 2020-09-03 20:12:38 -07:00
Chris Lu
f815fce81b
Merge pull request #1446 from taozix/master 
new filer store option to es7.
 2020-09-03 20:10:23 -07:00
Chris Lu
44b4ebf0ff filer: Redis cleanly delete directory 
fix https://github.com/chrislusf/seaweedfs/issues/1448
 2020-09-03 19:17:39 -07:00