Commit Graph

3340 Commits

Author SHA1 Message Date
Chris Lu
ccbdb38c89 s3: fix for listing objects if more than 1000 in the folder
many existing tools have max key set to 1000
2021-01-24 00:27:52 -08:00
Chris Lu
80b8692688 filer.sync: replicate outside of either cluster, only need to see filers 2021-01-24 00:01:44 -08:00
Chris Lu
f8dbb03bdd filer: append operation returns final file size 2021-01-23 04:19:49 -08:00
Chris Lu
759482e2ac filer: proxy all http methods to volume servers 2021-01-23 03:52:38 -08:00
Chris Lu
937cfacc01 filer: add "proxyToFileId" to reverse proxy to a volume server 2021-01-23 03:43:48 -08:00
Chris Lu
20ef3bb8d4 mount: use direct_io to avoid OS page cache
fix https://github.com/chrislusf/seaweedfs/issues/1752
2021-01-22 22:39:46 -08:00
Chris Lu
711c3f3939 filer: fix http status 206 setting
fix https://github.com/chrislusf/seaweedfs/issues/1753
2021-01-22 00:26:30 -08:00
Chris Lu
024c186a48 filer.meta.tail: output json format for easier parsing 2021-01-21 22:39:25 -08:00
Chris Lu
84f05787f8 mount: limit background requests and congestion threshold 2021-01-20 23:30:14 -08:00
Chris Lu
f98817cfe6 filer: support appending to a file 2021-01-20 13:40:32 -08:00
Chris Lu
1b8e3da295 refactoring 2021-01-20 12:59:54 -08:00
Chris Lu
1a3f859c58 filer store: postgres configuration adjust text 2021-01-19 18:12:07 -08:00
Chris Lu
b434f7e4e0 filer store: postgres adjust default config, add optioanl schema 2021-01-19 18:10:36 -08:00
Chris Lu
d5add83e85 filer store: add postgres2 2021-01-19 18:07:29 -08:00
Chris Lu
52a8f1470e filer store: add mysql2 2021-01-19 17:21:50 -08:00
Chris Lu
93b3adba98 fix bucket creation 2021-01-19 15:55:51 -08:00
Chris Lu
fa0c8d5283 fix error 2021-01-19 14:05:48 -08:00
Chris Lu
4c5b752b04 restructuring sql stores 2021-01-19 13:53:16 -08:00
Chris Lu
96354208c5 scaffold: add YugabyteDB 2021-01-19 12:35:14 -08:00
Chris Lu
ca8f793978 refactor: prepare for bucket specific tables 2021-01-19 12:34:58 -08:00
Chris Lu
90ce1eec19 add more input params 2021-01-19 11:32:55 -08:00
Chris Lu
f13c082583 add memsql to comments 2021-01-19 11:31:55 -08:00
Chris Lu
e439b65e38 fix test 2021-01-18 01:48:00 -08:00
Chris Lu
c7197470fd 2.21 2021-01-18 01:25:04 -08:00
Chris Lu
5a3386e39b Revert "Revert "mount: fake support for socket/block/character/fifo devices""
This reverts commit 3d5cb7eb86.
2021-01-18 01:15:07 -08:00
Chris Lu
67faa56920 Revert "Revert "mount: add more entry type""
This reverts commit 4085b79d38.
2021-01-18 01:14:58 -08:00
Chris Lu
7119e7b491 mount: ensure reading from dirty pages 2021-01-18 01:14:42 -08:00
Chris Lu
de876c795d minor fix 2021-01-18 01:14:27 -08:00
Chris Lu
4085b79d38 Revert "mount: add more entry type"
This reverts commit d7d907be27.
2021-01-18 00:19:19 -08:00
Chris Lu
3d5cb7eb86 Revert "mount: fake support for socket/block/character/fifo devices"
This reverts commit 61ef2d8658.
2021-01-18 00:18:57 -08:00
Chris Lu
61ef2d8658 mount: fake support for socket/block/character/fifo devices
to pass pjdfstest. A distributed file system can not really support these things anyway.
2021-01-17 23:51:10 -08:00
Chris Lu
d7d907be27 mount: add more entry type 2021-01-17 23:46:39 -08:00
Chris Lu
389426bbb7 s3: listing may repeat on the edge 2021-01-17 22:59:31 -08:00
Chris Lu
09f49d1c04 refactoring 2021-01-16 19:52:15 -08:00
Chris Lu
a4063a5437 add stream list directory entries 2021-01-15 23:56:24 -08:00
Chris Lu
01dc8a43ba fix rocksdb 2021-01-15 18:31:29 -08:00
Chris Lu
16ad74f477 go fmt 2021-01-14 23:11:27 -08:00
Chris Lu
f002e668de change limit to int64 in case of overflow 2021-01-14 23:10:37 -08:00
Chris Lu
19e52fd414 pass along hasMore 2021-01-14 22:49:29 -08:00
Chris Lu
c0bcf6f6e1 implement sql changes 2021-01-14 22:44:22 -08:00
Chris Lu
5d4568b91f implement elastic changes 2021-01-14 22:42:25 -08:00
Chris Lu
893cbc8482 implement c* changes 2021-01-14 22:38:34 -08:00
Chris Lu
5ef43b9b09 implement etcd changes 2021-01-14 22:35:56 -08:00
Chris Lu
54527f0326 implement hbase changes 2021-01-14 22:34:35 -08:00
Chris Lu
b5ceffe188 implement leveldb changes 2021-01-14 22:33:05 -08:00
Chris Lu
c64bfb0e2e implement mongodb changes 2021-01-14 22:28:51 -08:00
Chris Lu
34a846009d implement rocksdb changes 2021-01-14 22:24:26 -08:00
Chris Lu
d2ece29fff implement redis changes 2021-01-14 22:21:31 -08:00
Chris Lu
9a50dbcda0 chagned api 2021-01-14 22:18:38 -08:00
Chris Lu
e1c7bc66f9 change interface 2021-01-14 22:17:49 -08:00
Chris Lu
4aaa97c789 skip not found error in case of race condition 2021-01-14 19:56:14 -08:00
Chris Lu
d0826c2f62 mount: avoid showing "topics" folder
fix https://github.com/chrislusf/seaweedfs/issues/1325
2021-01-14 19:03:15 -08:00
Chris Lu
698f58f7c4 filer, s3: add http status 206 as late as possible
fix https://github.com/chrislusf/seaweedfs/issues/1746
2021-01-14 02:59:40 -08:00
Chris Lu
fd1d8a2a09 fix compilation 2021-01-14 00:06:15 -08:00
Chris Lu
ae812c07d6
Merge pull request #1743 from LazyDBA247-Anyvision/master
MYSQL - Better Connection Pool Control
2021-01-14 00:04:44 -08:00
Chris Lu
f20743bb38 remove WIP status 2021-01-13 22:21:34 -08:00
LazyDBA247-Anyvision
8eed763b97 better mysql connection pool management
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime)
to enable refresh of connections.
2021-01-14 08:14:21 +02:00
Chris Lu
f17aa1d06c skip directory loop if dropping the bucket 2021-01-13 13:49:04 -08:00
Chris Lu
ca73013453 leveldb3: ensure repeatable adding removing buckets 2021-01-13 13:48:48 -08:00
Chris Lu
e2c7e3fe6d better locks 2021-01-13 13:20:33 -08:00
Chris Lu
819a85f59a avoid viper concurrent access 2021-01-13 13:14:52 -08:00
Chris Lu
3fb2ed9093 filer.meta.tail: optionally submit metadata changes to ElasticSearch 2021-01-13 03:31:37 -08:00
Chris Lu
c5df2577f5 rename 2021-01-13 00:31:19 -08:00
Chris Lu
90c5077610 rename "weed watch" to "weed filer.meta.tail" 2021-01-12 18:48:01 -08:00
Chris Lu
5cd9c9e642 remove println 2021-01-12 18:47:38 -08:00
Chris Lu
def6d8e565 fix test 2021-01-12 12:32:54 -08:00
Chris Lu
1c7faf1100 Revert "close opened file"
This reverts commit 2880160113.
2021-01-12 11:07:14 -08:00
Chris Lu
2880160113 close opened file 2021-01-12 11:03:08 -08:00
Chris Lu
0ef43a23a7 go fmt 2021-01-12 02:30:12 -08:00
Chris Lu
629c9962e7 filer: add leveldb3
support dedicated leveldb instance for each bucket
2021-01-12 02:29:44 -08:00
Chris Lu
361043e6c1 filer store: leveldb2 fix nil entry error if not found 2021-01-12 02:28:57 -08:00
Chris Lu
cfb9342a15 avoid concurrent map updates to viper 2021-01-12 02:28:13 -08:00
Chris Lu
38d516251e Revert "mount: avoid fatal error"
This reverts commit a4f0bac9b7.
2021-01-11 13:50:15 -08:00
Chris Lu
a4f0bac9b7 mount: avoid fatal error 2021-01-11 13:26:59 -08:00
Chris Lu
8a7302ce37 s3: KeyCount is zero
fix https://github.com/chrislusf/seaweedfs/issues/1741
2021-01-11 03:50:45 -08:00
Chris Lu
2c7148cd55 redis cluster: do not enable route by latency by default 2021-01-11 02:43:53 -08:00
Chris Lu
acc0d5cd2c Revert "Revert "filer store: redis module upgrade""
This reverts commit dd0171e35c.
2021-01-11 02:30:19 -08:00
Chris Lu
dd0171e35c Revert "filer store: redis module upgrade"
This reverts commit 40161c9cec.
2021-01-11 02:09:18 -08:00
Chris Lu
394513f598 filer: ensure seamless meta data updates 2021-01-11 02:08:55 -08:00
Chris Lu
6e12a3a490 refactoring 2021-01-11 02:08:26 -08:00
Chris Lu
62ee484d12 refactoring 2021-01-11 00:03:13 -08:00
Chris Lu
1efb51ba84 filer: change to saveToFilerLimit from cacheToFilerLimit
short circuit saving small files to volume server
2021-01-10 23:14:46 -08:00
Chris Lu
f0d3b3bf93 2.20 2021-01-09 22:55:22 -08:00
Chris Lu
d3d3f2fb9b mount: default to 128 concurrent writers 2021-01-09 22:53:37 -08:00
Chris Lu
90b117acf1 update ccache version 2021-01-08 02:17:43 -08:00
Chris Lu
e2e0e30a51 filer: redis default to false for useReadOnly and routeByLatency 2021-01-08 01:12:44 -08:00
Chris Lu
40161c9cec filer store: redis module upgrade 2021-01-08 01:08:44 -08:00
Chris Lu
30efaa66c6 server: add option to disable master 2021-01-07 02:35:47 -08:00
Chris Lu
07f4703bfc
Merge pull request #1738 from qieqieplus/rocksdb
fix rocksdb crash when list directory
2021-01-07 02:26:41 -08:00
qieqieplus
0f18592315 alter style 2021-01-07 18:21:48 +08:00
qieqieplus
c7e7431fd3 fix seek lastkey may reach EOF 2021-01-07 17:07:56 +08:00
Chris Lu
e327385644 go fmt 2021-01-06 04:22:00 -08:00
Chris Lu
2b76854641 add "weed filer.cat" to read files directly from volume servers 2021-01-06 04:22:00 -08:00
Chris Lu
95ecf0c72f
Merge pull request #1735 from qieqieplus/rocksdb
ignore decode error for non-entry data
2021-01-06 02:22:18 -08:00
qieqieplus
0764fccde7 ignore decode error for non-entry data 2021-01-06 17:42:43 +08:00
Chris Lu
ab57767d4b
Merge pull request #1730 from qieqieplus/rocksdb
impl: TTL per entry for rocksdb; fix package name
2021-01-05 13:24:45 -08:00
qieqieplus
b2e50f602f
fix time comparison 2021-01-05 19:36:40 +08:00
qieqieplus
bbae13e415 impl: TTL per entry for rocksdb; fix package name 2021-01-05 17:58:40 +08:00
henry
b42c5b71c1 Seaweed custom header are not visible to Vue or javascript 2021-01-05 13:01:29 +08:00
henry
1309dfe612 Seaweed custom header are not visible to Vue or javascript 2021-01-05 12:58:46 +08:00
henry
942a30f04c remove sort 2021-01-05 12:24:32 +08:00
henry
a3c432dadc Merge branch 'master' of https://github.com/fuyouyshengwu/seaweedfs 2021-01-05 11:17:29 +08:00
qieqieplus
ebb223c190 fix for rocksdb kv 2021-01-04 21:21:46 +08:00
chenqieqie
40b3207f18 fix #1726 2021-01-04 18:48:55 +08:00
henry
97a94eddab Merge branch 'master' of https://github.com/fuyouyshengwu/seaweedfs 2021-01-04 14:50:01 +08:00
Chris Lu
2ce86f308e 2.19 2021-01-03 17:24:02 -08:00
Chris Lu
9c9ba3c209 nil related
related to https://github.com/chrislusf/seaweedfs/issues/1676
2021-01-03 12:25:58 -08:00
Chris Lu
4b5adc60b4 upload result changes to the right output variable 2021-01-03 01:44:22 -08:00
Chris Lu
039ad101b1 manifest file also need to detect compression
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753585046
2021-01-03 01:42:21 -08:00
Chris Lu
5f538c5f9d Merge branch 'add_rocksdb' 2021-01-03 00:48:56 -08:00
Chris Lu
bdd166c031 adjust readme 2021-01-03 00:47:58 -08:00
Chris Lu
a04d8927a8 add scaffold 2021-01-03 00:44:52 -08:00
Chris Lu
4250a3a675 upload: add read default collection from master
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753576071
2021-01-02 23:43:01 -08:00
Chris Lu
f4d87f8b9d works
performance seems about the same as leveldb for insert entries
2021-01-02 23:32:58 -08:00
Chris Lu
2c3c2c27d7 separate prefix from namePattern
fix https://github.com/chrislusf/seaweedfs/issues/1722
2021-01-01 20:23:23 -08:00
Peter Cai
cd2a9865ed filer: respect Content-Type on PUT
This is needed for the S3 gateway to conform to the S3 protocol -- the
server should always respect the Content-Type set by client.
2020-12-31 11:32:45 +08:00
Chris Lu
3433accb1b weed upload: add TTL 2020-12-30 15:46:07 -08:00
henry
14ddd15508 1 Distinguish between folders and common files, folders are listed first
2 Add some custom headers that need to be exposed
2020-12-31 00:03:22 +08:00
Chris Lu
adef4ddc87 mount: load security keys
fix https://github.com/chrislusf/seaweedfs/issues/1714
2020-12-29 11:51:38 -08:00
Chris Lu
89977123b6 shell: add shell.toml support
fix https://github.com/chrislusf/seaweedfs/issues/1664
2020-12-28 15:07:18 -08:00
Chris Lu
d2ead72918 2.18 2020-12-28 12:37:10 -08:00
Chris Lu
2e6cf541e7 filer: avoid deletion error when item is not found 2020-12-28 00:18:37 -08:00
Chris Lu
6d2c979999 s3: fix regression
fix https://github.com/chrislusf/seaweedfs/issues/1707
2020-12-27 21:09:45 -08:00
Chris Lu
90df52d001 s3: add command option "allowEmptyFolder" to filer and server 2020-12-27 21:09:06 -08:00
Chris Lu
da7e5aaa65 filer: optimize for less number of directory lookup
bottom up directory lookup
2020-12-27 20:46:17 -08:00
henry
7458ff9523 Solve the Chinese name disorder of download file 2020-12-28 11:28:33 +08:00
Chris Lu
561a2ff0bc s3: add option for "alllowEmptyFolder" 2020-12-27 10:50:27 -08:00
Chris Lu
aa020ee7e7 s3: restore V2 auth type
The added headers were included in calculating the signatures, failing the comparison
2020-12-26 22:01:16 -08:00
Chris Lu
da87f6b265 remove unused code 2020-12-26 15:21:12 -08:00
Chris Lu
0a7c5f85a9 filer: add namePattern to search in current folder 2020-12-26 15:05:31 -08:00
Chris Lu
515b7632d7 2.17 2020-12-25 23:58:22 -08:00
Chris Lu
0a067944cc filer: add retries during volume moving
fix https://github.com/chrislusf/seaweedfs/issues/1704
2020-12-25 02:32:55 -08:00
Chris Lu
090f85be4b s3: support config action Admin:bucket 2020-12-25 00:38:56 -08:00
Chris Lu
75613b2cc7 Merge branch 'add_hbase' 2020-12-24 12:13:05 -08:00
Chris Lu
0e016bc7bd hbase add ttl 2020-12-24 12:10:35 -08:00
Chris Lu
8e48a235e2 s3: avoid looping if the directory is empty
fix https://github.com/chrislusf/seaweedfs/issues/1701
2020-12-24 11:34:52 -08:00
Chris Lu
1620de08ae added more logs 2020-12-24 01:52:06 -08:00
Chris Lu
a09cd28986 add debug messages 2020-12-24 01:42:15 -08:00
Chris Lu
75c6edba9e filer: hbase add enabled flag 2020-12-24 00:19:16 -08:00
Chris Lu
9bf6c10505 fix prefix 2020-12-24 00:18:59 -08:00
Chris Lu
2fea8cfa0f fix compilation 2020-12-23 23:54:15 -08:00
Chris Lu
53bc1ea25b fix compilation 2020-12-23 23:53:46 -08:00
Chris Lu
b5e2be635a adjust for directory listing 2020-12-23 23:49:22 -08:00
Chris Lu
c4a202ec41 fix wrong column family 2020-12-23 23:23:05 -08:00
Chris Lu
c3d1b3b5aa hook up 2020-12-23 21:49:01 -08:00
Chris Lu
94e3757c08 fix 2020-12-23 21:45:16 -08:00
Chris Lu
64c48c9724 seems compiling 2020-12-23 21:44:53 -08:00
Chris Lu
15da5834e1 Merge branch 'master' into support_ssd_volume 2020-12-23 14:37:37 -08:00
Chris Lu
3be3635799 just log error when checking empty dir failed 2020-12-23 14:36:48 -08:00
Chris Lu
0ca9d89589 s3: break loop if error
fix #1701
2020-12-23 14:34:59 -08:00
Chris Lu
132f275d04 s3: log errors instead of stopping when checking isDirectoryAllEmpty has error 2020-12-23 12:21:21 -08:00
Chris Lu
db62090f88 s3: break loop if error
fix https://github.com/chrislusf/seaweedfs/issues/1701
2020-12-23 12:14:51 -08:00
Baptiste Mille-Mathias
385a4dec30 [shell] Clarify bucket replication setting order 2020-12-23 09:35:24 +01:00
Chris Lu
1c7e1295dc Merge branch 'master' into support_ssd_volume 2020-12-22 17:44:52 -08:00
Chris Lu
da134a2eb7 minor 2020-12-22 17:43:13 -08:00
Chris Lu
beb3b8ddb0 shell: change bucket.list to s3.bucket.list, same for create and delete 2020-12-22 17:40:55 -08:00
Chris Lu
4cf3176cbb filer first, master second 2020-12-22 17:34:44 -08:00
Chris Lu
a1009e8044 filer: add -defaultStoreDir so that filer.toml can be skipped
fix https://github.com/chrislusf/seaweedfs/issues/1659
2020-12-22 17:33:40 -08:00
Chris Lu
a0990b929d minor 2020-12-22 17:33:40 -08:00
Baptiste Mille-Mathias
0ed5345954 [shell] Add more information to bucket.create
State the replication setting that will be honor if not defined.
2020-12-22 17:18:59 +01:00
Chris Lu
97e3432dfe avoid wrong error
fix https://github.com/chrislusf/seaweedfs/issues/1691
2020-12-22 02:46:24 -08:00
Chris Lu
4f31c1bb94 go fmt 2020-12-22 02:34:08 -08:00
Chris Lu
6c4f32d173 remove unused code 2020-12-22 02:33:31 -08:00
Chris Lu
90d785a15f filer: redis, redis cluster, cassandra support super large directory 2020-12-22 02:26:05 -08:00
Chris Lu
fe46411cd4 handle directory listing edge cases 2020-12-21 23:37:43 -08:00
Chris Lu
c74bede730 handle listing directories 2020-12-21 23:19:05 -08:00
Chris Lu
adf8cb4000 filer: path-specific stores trim out common prefixes 2020-12-21 22:57:13 -08:00
Chris Lu
0823bde8d8 refactor: split into two files 2020-12-21 22:05:15 -08:00
Chris Lu
8ab70ccf76 refactor 2020-12-21 21:46:32 -08:00
Chris Lu
65cc6dc636 refactor to dedicated function 2020-12-21 21:18:34 -08:00
Chris Lu
488c2680e8 change default filer store root 2020-12-21 21:15:22 -08:00
Chris Lu
f5a19a87be filer: report error if the path specific store is not found
related https://github.com/chrislusf/seaweedfs/issues/1691
2020-12-21 12:43:52 -08:00
Chris Lu
6912bf94ae 2.16 2020-12-20 15:30:12 -08:00
Chris Lu
41c0f3ad24 filer: support path-specific filer store 2020-12-19 01:27:09 -08:00
Chris Lu
2260864b45 a cleaner fix 2020-12-18 10:39:30 -08:00
Chris Lu
b7fef8f693 fix when a volume server has only hdd volumes 2020-12-18 10:34:16 -08:00
Chris Lu
0d5683fb0e todo: load path-specific store from filer.toml 2020-12-18 03:05:42 -08:00
Chris Lu
3269fd7eaf filer: use store by path 2020-12-18 02:57:49 -08:00
Chris Lu
e605f1e001 refactor, add getActualStore() by path 2020-12-18 02:55:00 -08:00
Chris Lu
23903aa95b refactoring 2020-12-18 02:35:45 -08:00
Chris Lu
4e6b316913 assert FilerStoreWrapper is VirtualFilerStore 2020-12-18 02:17:06 -08:00
Chris Lu
9abf016af1 Merge branch 'master' into support_ssd_volume 2020-12-17 22:30:19 -08:00
Chris Lu
738c4fd203 fix logging when error
address https://github.com/chrislusf/seaweedfs/pull/1685#issuecomment-747845601
2020-12-17 19:37:46 -08:00
Chris Lu
d9e8479c06 adjust UI max count 2020-12-17 13:47:51 -08:00
Chris Lu
3cdf5945a2 adjust UI 2020-12-17 13:37:00 -08:00
Chris Lu
f696a2b2a7 assign volumes based on disk type 2020-12-17 13:25:05 -08:00
Chris Lu
daa8157fc2 Merge branch 'master' into support_ssd_volume 2020-12-17 13:05:20 -08:00
Chris Lu
e2076201d7 volume: avoid reprocessing the same volume
fix https://github.com/chrislusf/seaweedfs/issues/1682
2020-12-17 13:03:39 -08:00
Chris Lu
986cbdf7d9 Revert "Merge pull request #1683 from qieqieplus/master"
This reverts commit 8cb67952db, reversing
changes made to 200e56215a.
2020-12-17 12:46:42 -08:00
Chris Lu
f56e6d231e Revert "Merge pull request #1683 from qieqieplus/master"
This reverts commit 8cb67952db, reversing
changes made to 200e56215a.
2020-12-17 12:46:20 -08:00
Chris Lu
1bf22c0b5b go fmt 2020-12-16 09:14:05 -08:00
Chris Lu
1d88865869 passing disk type along 2020-12-16 09:10:14 -08:00
Chris Lu
23014b6810 Merge branch 'master' into support_ssd_volume 2020-12-16 08:26:51 -08:00
Chris Lu
b789767cd1 filer: return http status 499 for client closed connection
fix https://github.com/chrislusf/seaweedfs/issues/1684
2020-12-16 08:18:00 -08:00
chenqieqie
45f902a9b7 smaller critical section 2020-12-16 20:23:51 +08:00
chenqieqie
4e58a4f24e fix race condition when loading volumes concurrently 2020-12-16 18:49:10 +08:00
Chris Lu
3c7f9633eb show error a little bit more obviously
fix https://github.com/chrislusf/seaweedfs/issues/1678
2020-12-14 09:39:33 -08:00
Chris Lu
97dcae827b Merge branch 'master' into support_ssd_volume 2020-12-14 01:27:31 -08:00
Chris Lu
3d47c38262 collection.delete requires _default_ as the default empty collection name
fix https://github.com/chrislusf/seaweedfs/issues/1677
2020-12-14 01:05:20 -08:00
Chris Lu
2e8dba571b adjust volume server UI 2020-12-14 00:51:57 -08:00