Commit Graph

3500 Commits

Author SHA1 Message Date
Chris Lu
885ca34748 volume: fail fast if idx files are missing
fix https://github.com/chrislusf/seaweedfs/issues/1796
2021-02-11 00:44:40 -08:00
Chris Lu
a0e84c4fbc go fmt 2021-02-10 23:41:05 -08:00
Chris Lu
c959aaa19b mount: load directory based on correct filer path
fix https://github.com/chrislusf/seaweedfs/issues/1799
2021-02-10 23:14:18 -08:00
Konstantin Lebedev
0cfed8c3cb restart replication if RabbitMQ connection closed
https://github.com/google/go-cloud/issues/2958
2021-02-10 15:29:05 +05:00
Chris Lu
770393a48c volume: add capability to change disk type when moving a volume 2021-02-09 23:58:08 -08:00
Chris Lu
821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
Chris Lu
a833021132 fix refactoring left over 2021-02-07 23:03:03 -08:00
Chris Lu
5e4b5109dd 2.24 2021-02-07 22:29:43 -08:00
Chris Lu
15c60cbb26 close the grpc connection after 10 hours
related to https://github.com/chrislusf/seaweedfs/issues/1782
2021-02-07 03:50:01 -08:00
bingoohuang
7256902fb0 fix typo offset.ToAcutalOffset to offset.ToActualOffset 2021-02-07 12:11:51 +08:00
bingoohuang
94ea3bd3a5 renaming NeedleMapType to NeedleMapKind 2021-02-07 09:00:03 +08:00
Chris Lu
c3af72d950 fier store: fix elastic search regression
fix https://github.com/chrislusf/seaweedfs/issues/1774
2021-02-04 01:30:14 -08:00
Konstantin Lebedev
2fb4c60540 fix log warn 2021-02-03 21:34:32 +05:00
Konstantin Lebedev
24ec17219b RabbitMQ delay retry with Dead Letter Exchange
https://github.com/chrislusf/seaweedfs/issues/1773

https://github.com/google/go-cloud/issues/2952
2021-02-03 21:32:30 +05:00
Chris Lu
a331bbb3ae filer: should return 204 on DELETE to nonexistent file
related to

https://github.com/chrislusf/seaweedfs/issues/1776
https://github.com/chrislusf/seaweedfs/issues/1160
2021-02-03 00:40:31 -08:00
Chris Lu
5353e38469 S3: should return 204 on DELETE to nonexistent file
fix https://github.com/chrislusf/seaweedfs/issues/1776
2021-02-03 00:35:44 -08:00
Chris Lu
2396ac234c filer: fix elastic search pagination
possible fix for https://github.com/chrislusf/seaweedfs/issues/1774
2021-02-02 11:34:12 -08:00
Chris Lu
609daaf387 s3: DeleteMultipleObjectsHandler clean up leftover empty folders
fix https://github.com/chrislusf/seaweedfs/issues/1772
2021-02-01 10:49:17 -08:00
Chris Lu
d67ccb66c0 webdav: can start together with "weed server" or "weed filer" 2021-01-31 22:16:52 -08:00
Chris Lu
76430790b9 webdav: cache to version specific folder 2021-01-31 22:02:03 -08:00
Chris Lu
318a3d2efc 2.23 2021-01-31 20:18:06 -08:00
Chris Lu
4be51c0701 filer: leveldb and hbase may miss files when listing large directories more than 1024
fix https://github.com/chrislusf/seaweedfs/issues/1768
2021-01-31 20:11:44 -08:00
Chris Lu
1102ae32c4 fix concurrent map reads 2021-01-31 18:26:26 -08:00
Chris Lu
314dc1c957 filer: etcd store fix listing
fix https://github.com/chrislusf/seaweedfs/issues/1767
2021-01-30 13:38:44 -08:00
Chris Lu
d475c89fcc go fmt 2021-01-28 15:23:46 -08:00
Chris Lu
19295600f9 mount: change option name to volumeServerAccess, with publicUrl and filerProxy modes 2021-01-28 15:23:16 -08:00
Chris Lu
990fa69bfe add back AdjustedUrl() related code 2021-01-28 14:36:29 -08:00
Chris Lu
9292796ec2 fmt 2021-01-28 14:29:14 -08:00
Chris Lu
c2bf1a88ac delete from the deepest directory first when checking empty folders 2021-01-28 14:28:40 -08:00
Chris Lu
e9d8201925 s3: batch purge empty folders 2021-01-28 13:20:06 -08:00
Chris Lu
cf252fc0cd mount: report error when Flush()
The error in Release() is not working.

See https://github.com/jaderhs/libfuse/blob/master/FAQ

related to https://github.com/chrislusf/seaweedfs/issues/1765
2021-01-28 04:46:37 -08:00
Chris Lu
9a06c35da4 replicate: incremental sink only contains new and updated files
address da08402ba2
2021-01-28 02:39:22 -08:00
Chris Lu
da08402ba2 replicate: use creation time for local incremental file sink
related to https://github.com/chrislusf/seaweedfs/pull/1762
2021-01-28 02:17:41 -08:00
Chris Lu
822f1ade9d
Merge pull request #1762 from kmlebedev/backupsink
replication to create time date directory
2021-01-28 02:03:33 -08:00
Konstantin Lebedev
be1062b7fc rename in scaffold 2021-01-28 14:59:20 +05:00
Konstantin Lebedev
02fdc0a333 rename backup to local_incremental and use mtime 2021-01-28 14:56:13 +05:00
Chris Lu
5f72c388bc
Merge pull request #1764 from danlsgiga/handle-umask
Use provided umask on mount
2021-01-28 01:09:20 -08:00
Daniel Santos
d38613c49a match upstream 2021-01-27 20:56:41 -07:00
Chris Lu
b81956bcb5 mount: invalidate kernel cache when mounted to a filer path
fix https://github.com/chrislusf/seaweedfs/issues/1752#issuecomment-768178422
2021-01-27 10:28:37 -08:00
Konstantin Lebedev
6b54ff9912 replication to create time date directory 2021-01-27 15:01:33 +05:00
Chris Lu
f20ec82a28 update help message
see https://github.com/chrislusf/seaweedfs/issues/1758#issuecomment-768123224
2021-01-27 00:48:31 -08:00
Chris Lu
1045ab8b07 avoid error message 2021-01-27 00:21:59 -08:00
Chris Lu
0488c84bea adjust help message 2021-01-27 00:17:52 -08:00
Konstantin Lebedev
3634811408 replication to todays date directory 2021-01-27 12:45:58 +05:00
Chris Lu
5e07afb0f0
Merge pull request #1759 from kmlebedev/sink.local
replication to local disk storage
2021-01-26 22:31:54 -08:00
Chris Lu
b3f66199db shell: volume.fix.replication adds collectionPattern with wildcard characters
fix https://github.com/chrislusf/seaweedfs/issues/1758
2021-01-26 22:30:39 -08:00
Daniel Santos
0c64d60fdb Fix log message with correct mode 2021-01-26 21:45:52 -07:00
Daniel Santos
0aaaa9b470 Fix var assignment 2021-01-26 21:06:08 -07:00
Daniel Santos
5a761bb7a4 Enforce umask correctly 2021-01-26 20:42:10 -07:00
Chris Lu
ad2a20c8a5 notification add ack and nack 2021-01-26 11:08:44 -08:00
Konstantin Lebedev
612b7975a1 replication to local disk storage 2021-01-26 22:50:25 +05:00
Chris Lu
3a1d3d3413 mount: properly invalidate kernel node cache entry
fix https://github.com/chrislusf/seaweedfs/issues/1752
2021-01-26 02:50:53 -08:00
Chris Lu
cc1f3907ff filer.sync: default filerProxy to true 2021-01-25 10:14:42 -08:00
Chris Lu
62ebb917e0 mount: avoid the need to specify different cache directory for different mount
fix https://github.com/seaweedfs/seaweedfs-csi-driver/issues/11
2021-01-25 09:10:20 -08:00
Chris Lu
17f7c1c43f Merge branch 'mount_from_outside_cluster' 2021-01-24 19:03:02 -08:00
Chris Lu
00707ec00f mount: outsideContainerClusterMode proxy through filer
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
2021-01-24 19:01:58 -08:00
Chris Lu
9bcb28a3ea avoid possible nil reader
fix https://github.com/chrislusf/seaweedfs/issues/1754
2021-01-24 17:19:35 -08:00
Chris Lu
2c5eac5705 2.22 2021-01-24 14:06:09 -08:00
Chris Lu
6ca10725b8 Revert "mount: when outside cluster network, use filer as proxy to access volume servers"
This reverts commit 096e088d7b.
2021-01-24 03:15:19 -08:00
Chris Lu
096e088d7b mount: when outside cluster network, use filer as proxy to access volume servers 2021-01-24 01:41:38 -08:00
Chris Lu
ccbdb38c89 s3: fix for listing objects if more than 1000 in the folder
many existing tools have max key set to 1000
2021-01-24 00:27:52 -08:00
Chris Lu
80b8692688 filer.sync: replicate outside of either cluster, only need to see filers 2021-01-24 00:01:44 -08:00
Chris Lu
f8dbb03bdd filer: append operation returns final file size 2021-01-23 04:19:49 -08:00
Chris Lu
759482e2ac filer: proxy all http methods to volume servers 2021-01-23 03:52:38 -08:00
Chris Lu
937cfacc01 filer: add "proxyToFileId" to reverse proxy to a volume server 2021-01-23 03:43:48 -08:00
Chris Lu
20ef3bb8d4 mount: use direct_io to avoid OS page cache
fix https://github.com/chrislusf/seaweedfs/issues/1752
2021-01-22 22:39:46 -08:00
Chris Lu
711c3f3939 filer: fix http status 206 setting
fix https://github.com/chrislusf/seaweedfs/issues/1753
2021-01-22 00:26:30 -08:00
Chris Lu
024c186a48 filer.meta.tail: output json format for easier parsing 2021-01-21 22:39:25 -08:00
Chris Lu
84f05787f8 mount: limit background requests and congestion threshold 2021-01-20 23:30:14 -08:00
Chris Lu
f98817cfe6 filer: support appending to a file 2021-01-20 13:40:32 -08:00
Chris Lu
1b8e3da295 refactoring 2021-01-20 12:59:54 -08:00
Chris Lu
1a3f859c58 filer store: postgres configuration adjust text 2021-01-19 18:12:07 -08:00
Chris Lu
b434f7e4e0 filer store: postgres adjust default config, add optioanl schema 2021-01-19 18:10:36 -08:00
Chris Lu
d5add83e85 filer store: add postgres2 2021-01-19 18:07:29 -08:00
Chris Lu
52a8f1470e filer store: add mysql2 2021-01-19 17:21:50 -08:00
Chris Lu
93b3adba98 fix bucket creation 2021-01-19 15:55:51 -08:00
Chris Lu
fa0c8d5283 fix error 2021-01-19 14:05:48 -08:00
Chris Lu
4c5b752b04 restructuring sql stores 2021-01-19 13:53:16 -08:00
Chris Lu
96354208c5 scaffold: add YugabyteDB 2021-01-19 12:35:14 -08:00
Chris Lu
ca8f793978 refactor: prepare for bucket specific tables 2021-01-19 12:34:58 -08:00
Chris Lu
90ce1eec19 add more input params 2021-01-19 11:32:55 -08:00
Chris Lu
f13c082583 add memsql to comments 2021-01-19 11:31:55 -08:00
Chris Lu
e439b65e38 fix test 2021-01-18 01:48:00 -08:00
Chris Lu
c7197470fd 2.21 2021-01-18 01:25:04 -08:00
Chris Lu
5a3386e39b Revert "Revert "mount: fake support for socket/block/character/fifo devices""
This reverts commit 3d5cb7eb86.
2021-01-18 01:15:07 -08:00
Chris Lu
67faa56920 Revert "Revert "mount: add more entry type""
This reverts commit 4085b79d38.
2021-01-18 01:14:58 -08:00
Chris Lu
7119e7b491 mount: ensure reading from dirty pages 2021-01-18 01:14:42 -08:00
Chris Lu
de876c795d minor fix 2021-01-18 01:14:27 -08:00
Chris Lu
4085b79d38 Revert "mount: add more entry type"
This reverts commit d7d907be27.
2021-01-18 00:19:19 -08:00
Chris Lu
3d5cb7eb86 Revert "mount: fake support for socket/block/character/fifo devices"
This reverts commit 61ef2d8658.
2021-01-18 00:18:57 -08:00
Chris Lu
61ef2d8658 mount: fake support for socket/block/character/fifo devices
to pass pjdfstest. A distributed file system can not really support these things anyway.
2021-01-17 23:51:10 -08:00
Chris Lu
d7d907be27 mount: add more entry type 2021-01-17 23:46:39 -08:00
Chris Lu
389426bbb7 s3: listing may repeat on the edge 2021-01-17 22:59:31 -08:00
Chris Lu
09f49d1c04 refactoring 2021-01-16 19:52:15 -08:00
Chris Lu
a4063a5437 add stream list directory entries 2021-01-15 23:56:24 -08:00
Chris Lu
01dc8a43ba fix rocksdb 2021-01-15 18:31:29 -08:00
Chris Lu
16ad74f477 go fmt 2021-01-14 23:11:27 -08:00
Chris Lu
f002e668de change limit to int64 in case of overflow 2021-01-14 23:10:37 -08:00
Chris Lu
19e52fd414 pass along hasMore 2021-01-14 22:49:29 -08:00
Chris Lu
c0bcf6f6e1 implement sql changes 2021-01-14 22:44:22 -08:00
Chris Lu
5d4568b91f implement elastic changes 2021-01-14 22:42:25 -08:00
Chris Lu
893cbc8482 implement c* changes 2021-01-14 22:38:34 -08:00
Chris Lu
5ef43b9b09 implement etcd changes 2021-01-14 22:35:56 -08:00
Chris Lu
54527f0326 implement hbase changes 2021-01-14 22:34:35 -08:00
Chris Lu
b5ceffe188 implement leveldb changes 2021-01-14 22:33:05 -08:00
Chris Lu
c64bfb0e2e implement mongodb changes 2021-01-14 22:28:51 -08:00
Chris Lu
34a846009d implement rocksdb changes 2021-01-14 22:24:26 -08:00
Chris Lu
d2ece29fff implement redis changes 2021-01-14 22:21:31 -08:00
Chris Lu
9a50dbcda0 chagned api 2021-01-14 22:18:38 -08:00
Chris Lu
e1c7bc66f9 change interface 2021-01-14 22:17:49 -08:00
Chris Lu
4aaa97c789 skip not found error in case of race condition 2021-01-14 19:56:14 -08:00
Chris Lu
d0826c2f62 mount: avoid showing "topics" folder
fix https://github.com/chrislusf/seaweedfs/issues/1325
2021-01-14 19:03:15 -08:00
Chris Lu
698f58f7c4 filer, s3: add http status 206 as late as possible
fix https://github.com/chrislusf/seaweedfs/issues/1746
2021-01-14 02:59:40 -08:00
Chris Lu
fd1d8a2a09 fix compilation 2021-01-14 00:06:15 -08:00
Chris Lu
ae812c07d6
Merge pull request #1743 from LazyDBA247-Anyvision/master
MYSQL - Better Connection Pool Control
2021-01-14 00:04:44 -08:00
Chris Lu
f20743bb38 remove WIP status 2021-01-13 22:21:34 -08:00
LazyDBA247-Anyvision
8eed763b97 better mysql connection pool management
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime)
to enable refresh of connections.
2021-01-14 08:14:21 +02:00
Chris Lu
f17aa1d06c skip directory loop if dropping the bucket 2021-01-13 13:49:04 -08:00
Chris Lu
ca73013453 leveldb3: ensure repeatable adding removing buckets 2021-01-13 13:48:48 -08:00
Chris Lu
e2c7e3fe6d better locks 2021-01-13 13:20:33 -08:00
Chris Lu
819a85f59a avoid viper concurrent access 2021-01-13 13:14:52 -08:00
Chris Lu
3fb2ed9093 filer.meta.tail: optionally submit metadata changes to ElasticSearch 2021-01-13 03:31:37 -08:00
Chris Lu
c5df2577f5 rename 2021-01-13 00:31:19 -08:00
Chris Lu
90c5077610 rename "weed watch" to "weed filer.meta.tail" 2021-01-12 18:48:01 -08:00
Chris Lu
5cd9c9e642 remove println 2021-01-12 18:47:38 -08:00
Chris Lu
def6d8e565 fix test 2021-01-12 12:32:54 -08:00
Chris Lu
1c7faf1100 Revert "close opened file"
This reverts commit 2880160113.
2021-01-12 11:07:14 -08:00
Chris Lu
2880160113 close opened file 2021-01-12 11:03:08 -08:00
Chris Lu
0ef43a23a7 go fmt 2021-01-12 02:30:12 -08:00
Chris Lu
629c9962e7 filer: add leveldb3
support dedicated leveldb instance for each bucket
2021-01-12 02:29:44 -08:00
Chris Lu
361043e6c1 filer store: leveldb2 fix nil entry error if not found 2021-01-12 02:28:57 -08:00
Chris Lu
cfb9342a15 avoid concurrent map updates to viper 2021-01-12 02:28:13 -08:00
Chris Lu
38d516251e Revert "mount: avoid fatal error"
This reverts commit a4f0bac9b7.
2021-01-11 13:50:15 -08:00
Chris Lu
a4f0bac9b7 mount: avoid fatal error 2021-01-11 13:26:59 -08:00
Chris Lu
8a7302ce37 s3: KeyCount is zero
fix https://github.com/chrislusf/seaweedfs/issues/1741
2021-01-11 03:50:45 -08:00
Chris Lu
2c7148cd55 redis cluster: do not enable route by latency by default 2021-01-11 02:43:53 -08:00
Chris Lu
acc0d5cd2c Revert "Revert "filer store: redis module upgrade""
This reverts commit dd0171e35c.
2021-01-11 02:30:19 -08:00
Chris Lu
dd0171e35c Revert "filer store: redis module upgrade"
This reverts commit 40161c9cec.
2021-01-11 02:09:18 -08:00
Chris Lu
394513f598 filer: ensure seamless meta data updates 2021-01-11 02:08:55 -08:00
Chris Lu
6e12a3a490 refactoring 2021-01-11 02:08:26 -08:00
Chris Lu
62ee484d12 refactoring 2021-01-11 00:03:13 -08:00
Chris Lu
1efb51ba84 filer: change to saveToFilerLimit from cacheToFilerLimit
short circuit saving small files to volume server
2021-01-10 23:14:46 -08:00
Chris Lu
f0d3b3bf93 2.20 2021-01-09 22:55:22 -08:00
Chris Lu
d3d3f2fb9b mount: default to 128 concurrent writers 2021-01-09 22:53:37 -08:00
Chris Lu
90b117acf1 update ccache version 2021-01-08 02:17:43 -08:00
Chris Lu
e2e0e30a51 filer: redis default to false for useReadOnly and routeByLatency 2021-01-08 01:12:44 -08:00
Chris Lu
40161c9cec filer store: redis module upgrade 2021-01-08 01:08:44 -08:00
Chris Lu
30efaa66c6 server: add option to disable master 2021-01-07 02:35:47 -08:00
Chris Lu
07f4703bfc
Merge pull request #1738 from qieqieplus/rocksdb
fix rocksdb crash when list directory
2021-01-07 02:26:41 -08:00
qieqieplus
0f18592315 alter style 2021-01-07 18:21:48 +08:00
qieqieplus
c7e7431fd3 fix seek lastkey may reach EOF 2021-01-07 17:07:56 +08:00
Chris Lu
e327385644 go fmt 2021-01-06 04:22:00 -08:00
Chris Lu
2b76854641 add "weed filer.cat" to read files directly from volume servers 2021-01-06 04:22:00 -08:00
Chris Lu
95ecf0c72f
Merge pull request #1735 from qieqieplus/rocksdb
ignore decode error for non-entry data
2021-01-06 02:22:18 -08:00
qieqieplus
0764fccde7 ignore decode error for non-entry data 2021-01-06 17:42:43 +08:00
Chris Lu
ab57767d4b
Merge pull request #1730 from qieqieplus/rocksdb
impl: TTL per entry for rocksdb; fix package name
2021-01-05 13:24:45 -08:00
qieqieplus
b2e50f602f
fix time comparison 2021-01-05 19:36:40 +08:00
qieqieplus
bbae13e415 impl: TTL per entry for rocksdb; fix package name 2021-01-05 17:58:40 +08:00
henry
b42c5b71c1 Seaweed custom header are not visible to Vue or javascript 2021-01-05 13:01:29 +08:00
henry
1309dfe612 Seaweed custom header are not visible to Vue or javascript 2021-01-05 12:58:46 +08:00
henry
942a30f04c remove sort 2021-01-05 12:24:32 +08:00
henry
a3c432dadc Merge branch 'master' of https://github.com/fuyouyshengwu/seaweedfs 2021-01-05 11:17:29 +08:00
qieqieplus
ebb223c190 fix for rocksdb kv 2021-01-04 21:21:46 +08:00
chenqieqie
40b3207f18 fix #1726 2021-01-04 18:48:55 +08:00
henry
97a94eddab Merge branch 'master' of https://github.com/fuyouyshengwu/seaweedfs 2021-01-04 14:50:01 +08:00
Chris Lu
2ce86f308e 2.19 2021-01-03 17:24:02 -08:00
Chris Lu
9c9ba3c209 nil related
related to https://github.com/chrislusf/seaweedfs/issues/1676
2021-01-03 12:25:58 -08:00
Chris Lu
4b5adc60b4 upload result changes to the right output variable 2021-01-03 01:44:22 -08:00
Chris Lu
039ad101b1 manifest file also need to detect compression
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753585046
2021-01-03 01:42:21 -08:00
Chris Lu
5f538c5f9d Merge branch 'add_rocksdb' 2021-01-03 00:48:56 -08:00
Chris Lu
bdd166c031 adjust readme 2021-01-03 00:47:58 -08:00
Chris Lu
a04d8927a8 add scaffold 2021-01-03 00:44:52 -08:00
Chris Lu
4250a3a675 upload: add read default collection from master
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753576071
2021-01-02 23:43:01 -08:00
Chris Lu
f4d87f8b9d works
performance seems about the same as leveldb for insert entries
2021-01-02 23:32:58 -08:00
Chris Lu
2c3c2c27d7 separate prefix from namePattern
fix https://github.com/chrislusf/seaweedfs/issues/1722
2021-01-01 20:23:23 -08:00
Peter Cai
cd2a9865ed filer: respect Content-Type on PUT
This is needed for the S3 gateway to conform to the S3 protocol -- the
server should always respect the Content-Type set by client.
2020-12-31 11:32:45 +08:00
Chris Lu
3433accb1b weed upload: add TTL 2020-12-30 15:46:07 -08:00
henry
14ddd15508 1 Distinguish between folders and common files, folders are listed first
2 Add some custom headers that need to be exposed
2020-12-31 00:03:22 +08:00
Chris Lu
adef4ddc87 mount: load security keys
fix https://github.com/chrislusf/seaweedfs/issues/1714
2020-12-29 11:51:38 -08:00
Chris Lu
89977123b6 shell: add shell.toml support
fix https://github.com/chrislusf/seaweedfs/issues/1664
2020-12-28 15:07:18 -08:00
Chris Lu
d2ead72918 2.18 2020-12-28 12:37:10 -08:00
Chris Lu
2e6cf541e7 filer: avoid deletion error when item is not found 2020-12-28 00:18:37 -08:00
Chris Lu
6d2c979999 s3: fix regression
fix https://github.com/chrislusf/seaweedfs/issues/1707
2020-12-27 21:09:45 -08:00
Chris Lu
90df52d001 s3: add command option "allowEmptyFolder" to filer and server 2020-12-27 21:09:06 -08:00
Chris Lu
da7e5aaa65 filer: optimize for less number of directory lookup
bottom up directory lookup
2020-12-27 20:46:17 -08:00
henry
7458ff9523 Solve the Chinese name disorder of download file 2020-12-28 11:28:33 +08:00
Chris Lu
561a2ff0bc s3: add option for "alllowEmptyFolder" 2020-12-27 10:50:27 -08:00
Chris Lu
aa020ee7e7 s3: restore V2 auth type
The added headers were included in calculating the signatures, failing the comparison
2020-12-26 22:01:16 -08:00
Chris Lu
da87f6b265 remove unused code 2020-12-26 15:21:12 -08:00
Chris Lu
0a7c5f85a9 filer: add namePattern to search in current folder 2020-12-26 15:05:31 -08:00
Chris Lu
515b7632d7 2.17 2020-12-25 23:58:22 -08:00
Chris Lu
0a067944cc filer: add retries during volume moving
fix https://github.com/chrislusf/seaweedfs/issues/1704
2020-12-25 02:32:55 -08:00
Chris Lu
090f85be4b s3: support config action Admin:bucket 2020-12-25 00:38:56 -08:00
Chris Lu
75613b2cc7 Merge branch 'add_hbase' 2020-12-24 12:13:05 -08:00
Chris Lu
0e016bc7bd hbase add ttl 2020-12-24 12:10:35 -08:00
Chris Lu
8e48a235e2 s3: avoid looping if the directory is empty
fix https://github.com/chrislusf/seaweedfs/issues/1701
2020-12-24 11:34:52 -08:00
Chris Lu
1620de08ae added more logs 2020-12-24 01:52:06 -08:00
Chris Lu
a09cd28986 add debug messages 2020-12-24 01:42:15 -08:00
Chris Lu
75c6edba9e filer: hbase add enabled flag 2020-12-24 00:19:16 -08:00
Chris Lu
9bf6c10505 fix prefix 2020-12-24 00:18:59 -08:00
Chris Lu
2fea8cfa0f fix compilation 2020-12-23 23:54:15 -08:00
Chris Lu
53bc1ea25b fix compilation 2020-12-23 23:53:46 -08:00
Chris Lu
b5e2be635a adjust for directory listing 2020-12-23 23:49:22 -08:00
Chris Lu
c4a202ec41 fix wrong column family 2020-12-23 23:23:05 -08:00
Chris Lu
c3d1b3b5aa hook up 2020-12-23 21:49:01 -08:00
Chris Lu
94e3757c08 fix 2020-12-23 21:45:16 -08:00
Chris Lu
64c48c9724 seems compiling 2020-12-23 21:44:53 -08:00
Chris Lu
15da5834e1 Merge branch 'master' into support_ssd_volume 2020-12-23 14:37:37 -08:00
Chris Lu
3be3635799 just log error when checking empty dir failed 2020-12-23 14:36:48 -08:00
Chris Lu
0ca9d89589 s3: break loop if error
fix #1701
2020-12-23 14:34:59 -08:00
Chris Lu
132f275d04 s3: log errors instead of stopping when checking isDirectoryAllEmpty has error 2020-12-23 12:21:21 -08:00
Chris Lu
db62090f88 s3: break loop if error
fix https://github.com/chrislusf/seaweedfs/issues/1701
2020-12-23 12:14:51 -08:00
Baptiste Mille-Mathias
385a4dec30 [shell] Clarify bucket replication setting order 2020-12-23 09:35:24 +01:00
Chris Lu
1c7e1295dc Merge branch 'master' into support_ssd_volume 2020-12-22 17:44:52 -08:00
Chris Lu
da134a2eb7 minor 2020-12-22 17:43:13 -08:00
Chris Lu
beb3b8ddb0 shell: change bucket.list to s3.bucket.list, same for create and delete 2020-12-22 17:40:55 -08:00
Chris Lu
4cf3176cbb filer first, master second 2020-12-22 17:34:44 -08:00
Chris Lu
a1009e8044 filer: add -defaultStoreDir so that filer.toml can be skipped
fix https://github.com/chrislusf/seaweedfs/issues/1659
2020-12-22 17:33:40 -08:00
Chris Lu
a0990b929d minor 2020-12-22 17:33:40 -08:00
Baptiste Mille-Mathias
0ed5345954 [shell] Add more information to bucket.create
State the replication setting that will be honor if not defined.
2020-12-22 17:18:59 +01:00
Chris Lu
97e3432dfe avoid wrong error
fix https://github.com/chrislusf/seaweedfs/issues/1691
2020-12-22 02:46:24 -08:00
Chris Lu
4f31c1bb94 go fmt 2020-12-22 02:34:08 -08:00
Chris Lu
6c4f32d173 remove unused code 2020-12-22 02:33:31 -08:00
Chris Lu
90d785a15f filer: redis, redis cluster, cassandra support super large directory 2020-12-22 02:26:05 -08:00
Chris Lu
fe46411cd4 handle directory listing edge cases 2020-12-21 23:37:43 -08:00
Chris Lu
c74bede730 handle listing directories 2020-12-21 23:19:05 -08:00
Chris Lu
adf8cb4000 filer: path-specific stores trim out common prefixes 2020-12-21 22:57:13 -08:00
Chris Lu
0823bde8d8 refactor: split into two files 2020-12-21 22:05:15 -08:00
Chris Lu
8ab70ccf76 refactor 2020-12-21 21:46:32 -08:00
Chris Lu
65cc6dc636 refactor to dedicated function 2020-12-21 21:18:34 -08:00
Chris Lu
488c2680e8 change default filer store root 2020-12-21 21:15:22 -08:00
Chris Lu
f5a19a87be filer: report error if the path specific store is not found
related https://github.com/chrislusf/seaweedfs/issues/1691
2020-12-21 12:43:52 -08:00
Chris Lu
6912bf94ae 2.16 2020-12-20 15:30:12 -08:00
Chris Lu
41c0f3ad24 filer: support path-specific filer store 2020-12-19 01:27:09 -08:00
Chris Lu
2260864b45 a cleaner fix 2020-12-18 10:39:30 -08:00
Chris Lu
b7fef8f693 fix when a volume server has only hdd volumes 2020-12-18 10:34:16 -08:00
Chris Lu
0d5683fb0e todo: load path-specific store from filer.toml 2020-12-18 03:05:42 -08:00
Chris Lu
3269fd7eaf filer: use store by path 2020-12-18 02:57:49 -08:00
Chris Lu
e605f1e001 refactor, add getActualStore() by path 2020-12-18 02:55:00 -08:00
Chris Lu
23903aa95b refactoring 2020-12-18 02:35:45 -08:00
Chris Lu
4e6b316913 assert FilerStoreWrapper is VirtualFilerStore 2020-12-18 02:17:06 -08:00
Chris Lu
9abf016af1 Merge branch 'master' into support_ssd_volume 2020-12-17 22:30:19 -08:00
Chris Lu
738c4fd203 fix logging when error
address https://github.com/chrislusf/seaweedfs/pull/1685#issuecomment-747845601
2020-12-17 19:37:46 -08:00
Chris Lu
d9e8479c06 adjust UI max count 2020-12-17 13:47:51 -08:00
Chris Lu
3cdf5945a2 adjust UI 2020-12-17 13:37:00 -08:00
Chris Lu
f696a2b2a7 assign volumes based on disk type 2020-12-17 13:25:05 -08:00
Chris Lu
daa8157fc2 Merge branch 'master' into support_ssd_volume 2020-12-17 13:05:20 -08:00
Chris Lu
e2076201d7 volume: avoid reprocessing the same volume
fix https://github.com/chrislusf/seaweedfs/issues/1682
2020-12-17 13:03:39 -08:00
Chris Lu
986cbdf7d9 Revert "Merge pull request #1683 from qieqieplus/master"
This reverts commit 8cb67952db, reversing
changes made to 200e56215a.
2020-12-17 12:46:42 -08:00
Chris Lu
f56e6d231e Revert "Merge pull request #1683 from qieqieplus/master"
This reverts commit 8cb67952db, reversing
changes made to 200e56215a.
2020-12-17 12:46:20 -08:00
Chris Lu
1bf22c0b5b go fmt 2020-12-16 09:14:05 -08:00
Chris Lu
1d88865869 passing disk type along 2020-12-16 09:10:14 -08:00
Chris Lu
23014b6810 Merge branch 'master' into support_ssd_volume 2020-12-16 08:26:51 -08:00
Chris Lu
b789767cd1 filer: return http status 499 for client closed connection
fix https://github.com/chrislusf/seaweedfs/issues/1684
2020-12-16 08:18:00 -08:00
chenqieqie
45f902a9b7 smaller critical section 2020-12-16 20:23:51 +08:00
chenqieqie
4e58a4f24e fix race condition when loading volumes concurrently 2020-12-16 18:49:10 +08:00
Chris Lu
3c7f9633eb show error a little bit more obviously
fix https://github.com/chrislusf/seaweedfs/issues/1678
2020-12-14 09:39:33 -08:00
Chris Lu
97dcae827b Merge branch 'master' into support_ssd_volume 2020-12-14 01:27:31 -08:00
Chris Lu
3d47c38262 collection.delete requires _default_ as the default empty collection name
fix https://github.com/chrislusf/seaweedfs/issues/1677
2020-12-14 01:05:20 -08:00
Chris Lu
2e8dba571b adjust volume server UI 2020-12-14 00:51:57 -08:00
Chris Lu
bf5873022d Merge branch 'master' into support_ssd_volume 2020-12-14 00:32:17 -08:00
Chris Lu
b7e3ca9172
Merge pull request #1650 from kmlebedev/masterServerEnableUI
security master use access.ui
2020-12-14 00:31:51 -08:00
Chris Lu
021358749b better detect IP address
following advice from https://github.com/chrislusf/seaweedfs/issues/1671
2020-12-14 00:30:20 -08:00
Konstantin Lebedev
c4459249ef ui interface is always on 2020-12-14 13:24:41 +05:00
Chris Lu
5cffddebb9 add logging 2020-12-14 00:11:52 -08:00
Chris Lu
933f2f4cfd find location with matching disk type 2020-12-13 23:39:00 -08:00
Chris Lu
94525aa0fd allocate volume by disk type 2020-12-13 23:08:21 -08:00
Chris Lu
f6a419c26c disk type configurable for each folder 2020-12-13 22:49:56 -08:00
Chris Lu
7c52a35942 refactor 2020-12-13 22:29:52 -08:00
Chris Lu
c45d6a865b Merge branch 'master' into support_ssd_volume 2020-12-13 22:14:18 -08:00
Chris Lu
38fc89d041 filer store: sql update if any insert error happens
fix https://github.com/chrislusf/seaweedfs/issues/1673
2020-12-13 20:49:44 -08:00
Chris Lu
23280257df sql put kv: avoid unnecessary update 2020-12-13 19:45:47 -08:00
Chris Lu
a9db24cd05 master allocate volumes if ssd type runs out 2020-12-13 19:44:57 -08:00
Chris Lu
ac4d4a65af Update Makefile 2020-12-13 12:34:58 -08:00
Chris Lu
8baba93fce rename parameter name to "disk" 2020-12-13 12:06:21 -08:00
Chris Lu
51eadaf2b6 rename parameter name to "disk" 2020-12-13 12:05:31 -08:00
Chris Lu
0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu
715b199eeb fix tests 2020-12-13 04:14:50 -08:00
Chris Lu
66cd219b15 Merge branch 'master' into support_ssd_volume 2020-12-13 04:08:36 -08:00
Chris Lu
813453f3e1 2.15 2020-12-13 03:45:35 -08:00
Chris Lu
17372ac3da balance ssd volumes 2020-12-13 03:40:33 -08:00
Chris Lu
d156c74ec0 volume server set volume type and heartbeat to the master 2020-12-13 03:11:24 -08:00
Chris Lu
e9cd798bd3 adding volume type 2020-12-13 00:58:58 -08:00
Chris Lu
16cd6fb278 fix variable name 2020-12-12 21:19:40 -08:00
Chris Lu
5c465293e9 correctly determine whether a folder is empty
avoid edge cases that deleting the folder if previous 32 directories are all empty

early terminate if one file is found
2020-12-12 16:19:29 -08:00
Chris Lu
14910d035c the pagination size was too big for recursive deletion 2020-12-12 13:26:10 -08:00
Chris Lu
f930c713fc more efficient recursion 2020-12-12 13:25:19 -08:00
Chris Lu
316d1b4e69 refactor APIs 2020-12-12 12:42:53 -08:00
Chris Lu
03637d6f57 s3: move "delete-directory-if-empty" to read time
move "delete-directory-if-empty" to read time instead of entry deletion time

the listing speed for a s3 bucket folder will slow down if it has many sub folders

related to 0d345ac97d

fix https://github.com/chrislusf/seaweedfs/issues/1647

fix https://github.com/chrislusf/seaweedfs/issues/1670
2020-12-12 03:38:34 -08:00
Chris Lu
37075a414d adjust logs 2020-12-12 03:33:57 -08:00
Chris Lu
ae655033ac adjust logging 2020-12-11 16:57:53 -08:00
Chris Lu
d986c7196d use append time instead of filer's own modification time
fix https://github.com/chrislusf/seaweedfs/issues/1669
2020-12-11 16:55:18 -08:00
Chris Lu
3fedfec1e7 check cross device rename error 2020-12-10 23:50:32 -08:00
Chris Lu
c2f18a10cb minor 2020-12-10 22:23:22 -08:00
Chris Lu
4b0c2a846b add logs 2020-12-10 19:55:28 -08:00
Chris Lu
83078ac6ce filer: change to /etc/seaweedfs folder on filer
fix https://github.com/chrislusf/seaweedfs/issues/1666
2020-12-10 11:11:02 -08:00
Chris Lu
26731694f8 s3: use static configuration by default
So that users can still use the previous configuration files.

If leave it empty, s3 will try to use the version from filer
2020-12-10 00:59:04 -08:00
Chris Lu
765b3ef1be save /etc/iam/identity.json inside filer store 2020-12-10 00:15:22 -08:00
Chris Lu
97c942b5ee paginate through large directories 2020-12-09 23:23:38 -08:00
Chris Lu
02a160c3fd s3: recursively iterate all sub folders
fix https://github.com/chrislusf/seaweedfs/issues/1656
2020-12-09 16:47:34 -08:00