mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-12 08:42:01 +08:00
Code Issues Packages Projects Releases Wiki Activity
5,482 Commits 32 Branches 296 Tags 150 MiB
319743d330
Commit Graph

3500 Commits

Author SHA1 Message Date
Konstantin Lebedev
bb3af2d70b no directoriesWithDeletion 2021-03-10 19:52:41 +05:00
Konstantin Lebedev
15b018da34 s3 delete-objects pass OK for fail to delete non-empty folder 2021-03-10 19:41:35 +05:00
Konstantin Lebedev
348e21a08c add comments 2021-03-10 14:42:39 +05:00
Konstantin Lebedev
831953c55c allowed wildcard domain 2021-03-10 14:02:13 +05:00
Konstantin Lebedev
4bf93d6e63 comma-separated 2021-03-10 12:43:13 +05:00
Konstantin Lebedev
0e02f7e258 comma-separated SSL certificate common names 2021-03-10 12:42:44 +05:00
Chris Lu
737bde5ab7 refactoring, remove duplicated code 2021-03-09 23:23:01 -08:00
Chris Lu
5ba4b479f8 properly lock file.entry object 
fix https://github.com/chrislusf/seaweedfs/issues/1882
 2021-03-09 23:08:38 -08:00
Chris Lu
5be83dd4a9 add comments 2021-03-09 14:13:48 -08:00
Chris Lu
9d8ca513d9 refactor 2021-03-09 13:21:26 -08:00
Chris Lu
f5b5e4da2d 2.31 2021-03-09 12:52:16 -08:00
Chris Lu
828f6e9f4d volume: auto add missing vif files 
fix https://github.com/chrislusf/seaweedfs/issues/1878
 2021-03-09 12:09:32 -08:00
Chris Lu
387c6f4218 Revert "delete the folder object in multi_object_delete" 
This reverts commit 2e89c8c9ae.
 2021-03-09 10:07:27 -08:00
qieqieplus
5b16820924 fix: collection stats won't update if all volumes expired at same iteration 2021-03-09 18:49:45 +08:00
Konstantin Lebedev
190fada1ef TLS allowed commonNames 2021-03-08 21:39:44 +05:00
Konstantin Lebedev
c6d3735605 permitCommonNames 
https://github.com/chrislusf/seaweedfs/issues/1841
https://jbrandhorst.com/post/grpc-auth/
 2021-03-08 13:16:17 +05:00
Chris Lu
d888e9a90f 2.30 2021-03-07 15:01:19 -08:00
Chris Lu
60686a0bb7 make tcp optional 2021-03-07 14:45:36 -08:00
Chris Lu
3b2a9c98ef fix nil 2021-03-07 11:45:35 -08:00
Chris Lu
df72dc206d stats collect chunk upload retry count 2021-03-07 11:26:15 -08:00
Chris Lu
726edab054 avoid nil when closing an index 
fix https://github.com/chrislusf/seaweedfs/issues/1870
 2021-03-07 11:03:09 -08:00
Chris Lu
bdfed16d42 avoid nil exception 
fix https://github.com/chrislusf/seaweedfs/issues/1869
 2021-03-07 01:49:06 -08:00
Chris Lu
1bd880dcdb adds tcp writes benchmark 2021-03-06 14:26:27 -08:00
Chris Lu
9d402ebe9f refactoring 2021-03-06 14:26:27 -08:00
Chris Lu
e9eddfb8a2 remove wrong writes 2021-03-06 14:26:27 -08:00
Chris Lu
8363be8548 trim out trailing return character 2021-03-06 14:26:27 -08:00
Chris Lu
38fc200e56 CRCWriter consistent with CRC 2021-03-06 14:26:27 -08:00
Chris Lu
1444e9d275 migrated multi host connection pool from godropbox package 
removing unneeded dependencies, which involved etcd versions.
 2021-03-06 14:26:27 -08:00
Patrick Schmidt
7413d59750 Fix EC shard count logic 
This fixes the calculation of the amount of EC shards a node holds.
Previously a global counter was increased, but also used inside the
loop to apply disk usage deltas. This led to wrong absolute numbers.
The fix is to apply only deltas of single EC shards per iteration.
 2021-03-05 12:50:58 +01:00
Chris Lu
400de380f4 volume server: support tcp direct put/get/delete 2021-03-05 02:29:38 -08:00
Chris Lu
2e89c8c9ae delete the folder object in multi_object_delete 
fix https://github.com/chrislusf/seaweedfs/issues/1857
 2021-03-04 14:03:40 -08:00
Chris Lu
c5cb346a08 fix compilation error 2021-03-04 12:36:20 -08:00
Chris Lu
f5cc96f289 skip checking master peers if not starting master 2021-03-04 12:31:26 -08:00
Chris Lu
40ff30b83f adjust help message 2021-03-03 02:12:46 -08:00
Chris Lu
c0842fe99f add filer.meta.backup command 2021-03-03 02:02:29 -08:00
Chris Lu
0c5c51eb98 adjust help message 2021-03-02 23:07:39 -08:00
Chris Lu
bd727b7b53 go fmt 2021-03-02 20:59:56 -08:00
Chris Lu
e982b06bcd support IPv6 2021-03-02 20:59:39 -08:00
Chris Lu
bcf32591b7 reduce possibility of nil entry 2021-03-02 13:33:56 -08:00
Chris Lu
be9c7c21ec reduce possibility of nil file entry 2021-03-02 11:17:17 -08:00
Chris Lu
02e146f3e1 return err 2021-03-02 08:54:18 -08:00
Chris Lu
5511722420 s3: list permission is needed to list my buckets 
fix https://github.com/chrislusf/seaweedfs/issues/1837
 2021-03-01 12:41:55 -08:00
Chris Lu
c46d16b2f3 skip network error when remote volumes start 2021-03-01 01:20:06 -08:00
Chris Lu
6e43e8ce17 add logs for remote file loading 2021-03-01 00:48:59 -08:00
Chris Lu
015d16f43f add vif file versions in case loading superblock fails 2021-03-01 00:48:30 -08:00
Chris Lu
d680676d45 skip already loaded backends 2021-03-01 00:47:03 -08:00
Chris Lu
540441fd38 go fmt 2021-02-28 20:34:14 -08:00
Chris Lu
c9722dceb2 adjust log level 2021-02-28 20:26:55 -08:00
Chris Lu
1b1c018165 adjust text 2021-02-28 19:02:43 -08:00
Chris Lu
2cca07b44c cloud tier to non-AWS s3 gateways 2021-02-28 18:59:09 -08:00
Chris Lu
f29ae0db76 2.29 2021-02-28 18:08:26 -08:00
bingoohuang
5cdff56731 set default env prefix to WEED_ 2021-03-01 09:35:23 +08:00
Chris Lu
e52c94640e filer.backup: added to replace filer.replicate 2021-02-28 16:22:27 -08:00
Chris Lu
014a31d11a minor 2021-02-28 16:21:09 -08:00
Chris Lu
f2fcb77808 local file sink: create backup files with permission 0755 2021-02-28 16:20:47 -08:00
Chris Lu
984fdd6192 always use non bucket prefixing url 2021-02-28 16:20:13 -08:00
Chris Lu
9abb041763 filer source: support filerProxy mode 2021-02-28 16:19:47 -08:00
Chris Lu
678c54d705 data sink: add incremental mode 2021-02-28 16:19:03 -08:00
Chris Lu
4ff2c5c4c9 rename file 2021-02-28 16:14:21 -08:00
Chris Lu
3b76a51f5f add help message 2021-02-27 12:15:49 -08:00
Chris Lu
70434df105 use "options" 2021-02-27 12:12:53 -08:00
Chris Lu
d3c31c69a7 avoid confusion: conf and config are too similar 2021-02-27 12:08:09 -08:00
Chris Lu
543c8daa6d
Merge pull request #1844 from kmlebedev/processRangeRequest 2021-02-26 09:18:22 -08:00
Patrick Schmidt
5f7b024891 Show the real disk usage in stats calls 
Currently the file size of only one volume location is taken into
account in the stats. This commit multiplies the disk usages by the
amount of nodes holding a replica of the volume.
This will yield the expected amount of disk usage and matches the
total size calculations from before.
 2021-02-26 13:58:40 +01:00
Konstantin Lebedev
ee21c0042e log error write entry 2021-02-26 14:18:01 +05:00
Chris Lu
7ba75e3d5a filer: do not return no content for empty files 
fix https://github.com/chrislusf/seaweedfs/issues/1831
fix https://github.com/chrislusf/seaweedfs/issues/1830
 2021-02-23 12:25:27 -08:00
Chris Lu
9edd964627 volume.tier.move: avoid repeated move for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-784139348
 2021-02-23 03:49:14 -08:00
Chris Lu
37f104f88f 2.28 2021-02-22 22:54:34 -08:00
Chris Lu
90cdf9dcac avoid conflict with "weed scaffold -config=xxx" 2021-02-22 16:57:27 -08:00
Chris Lu
72b0d9d8c4 avoid unnecessary user home checking 2021-02-22 16:50:56 -08:00
Chris Lu
2270737344 volume: avoid fixed vacuum timeout for large volumes 
1GB for 3 minutes, about 5.7MB/s
 2021-02-22 12:52:37 -08:00
Chris Lu
44bdfb2d15 filer: avoid encryption and compression at the same time 
fix https://github.com/chrislusf/seaweedfs/issues/1828
 2021-02-22 12:22:49 -08:00
Chris Lu
62191b08ea disk type support custom tags 2021-02-22 02:03:12 -08:00
Chris Lu
30b30b8fe0 volume.tier.move: passing non-empty disk type 2021-02-22 01:59:03 -08:00
Chris Lu
5da63e045e avoid moving to another server with the same volume id 2021-02-22 01:44:18 -08:00
Chris Lu
6a4546d2c0 shell: add volume.tier.move 2021-02-22 01:30:07 -08:00
Chris Lu
1c233ad986 refactoring 2021-02-22 00:28:42 -08:00
Chris Lu
151c281f36 2.27 2021-02-21 19:29:27 -08:00
Chris Lu
258e93bc86
Merge pull request #1823 from bingoohuang/master 
feature: support command line arguments in a configuration file
 2021-02-20 22:35:18 -08:00
Chris Lu
03c643aa6b fix test 
fileSize need to be divided by 8
 2021-02-20 21:19:21 -08:00
Chris Lu
7635f6b9fa disk file avoid file.Stat() 2021-02-20 20:06:06 -08:00
bingoo
1af6c96cd0 fix log help 2021-02-21 10:46:01 +08:00
Chris Lu
a2383b3b12 Revert "Revert "avoid file.Stat()"" 
This reverts commit 7ef4c24f18.
 2021-02-20 12:42:09 -08:00
Chris Lu
7ef4c24f18 Revert "avoid file.Stat()" 
This reverts commit 98c93ca465.
 2021-02-20 12:39:33 -08:00
Chris Lu
1ad3200094 skip seek() when index file writes 2021-02-20 12:39:25 -08:00
Chris Lu
98c93ca465 avoid file.Stat() 
avoid one Syscall, but did not help on performance though
 2021-02-20 09:44:17 -08:00
bingoohuang
ee082ae402 revert comment temporarily 2021-02-20 16:59:38 +08:00
bingoohuang
ee7cdf3668 revert ErrXyz to ErrorXyz temporarily. 2021-02-20 16:58:08 +08:00
bingoohuang
50df484d86 support command line arguments in a configuration file so we can weed master -config=master.conf, weed volume -config=volume.conf and etc. 2021-02-20 16:52:57 +08:00
bingoohuang
f69356f589
Merge pull request #3 from chrislusf/master 2021-02-20 16:45:02 +08:00
Chris Lu
565f7a6e72 Update data_node.go 2021-02-19 14:22:36 -08:00
Chris Lu
a37473ae60 add back volume ids 
address https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-782339576
 2021-02-19 14:22:12 -08:00
Chris Lu
a1210d1e8d Revert "volume: avoid file.stat(), file.seek() if possible during writes" 
This reverts commit c78409a598.
 2021-02-19 03:56:27 -08:00
Chris Lu
b961cd6208 add WIP message 2021-02-19 03:39:19 -08:00
Chris Lu
f465d63b5d grpc should fail when heart beating to master 
fix https://github.com/chrislusf/seaweedfs/issues/1820
 2021-02-19 03:37:29 -08:00
Chris Lu
c576ad04ac fix volume server display for volumes 2021-02-19 01:38:56 -08:00
bingoohuang
7ffe736d20 renaming to ErrXyz 2021-02-19 15:59:55 +08:00
Chris Lu
c78409a598 volume: avoid file.stat(), file.seek() if possible during writes 2021-02-18 23:22:54 -08:00
Chris Lu
29575dadc5 remove extra files 2021-02-18 19:14:58 -08:00
Chris Lu
73958e357d add descriptive error if no free volumes 2021-02-18 19:10:20 -08:00
Chris Lu
776f497469 filer: fs.configure should try to read from entry.content also 
related to https://github.com/chrislusf/seaweedfs/issues/1792
 2021-02-18 17:07:02 -08:00
Chris Lu
e1992c83de webdav adds disktype 2021-02-18 15:45:44 -08:00
Chris Lu
c2ad6f1047 webdav add replication setting 
fix https://github.com/chrislusf/seaweedfs/issues/1817
 2021-02-18 12:15:09 -08:00
Chris Lu
816ef0ed30 add more help message 
fix https://github.com/chrislusf/seaweedfs/issues/1816
 2021-02-18 10:34:49 -08:00
bingoohuang
352ac2f271 Merge remote-tracking branch 'origin/master' 2021-02-18 14:05:51 +08:00
bingoohuang
eab6e31d34 use backticks instead of double quotes to avoid escaped additionally in regex 2021-02-18 14:05:28 +08:00
Chris Lu
3575d41009 go fmt 2021-02-17 20:57:08 -08:00
Chris Lu
6daa932f5c refactoring to get master function, instead of passing master values directly 
this will enable retrying later
 2021-02-17 20:55:55 -08:00
Chris Lu
dd9f3a0104 add sleep between upload retries 2021-02-17 20:55:11 -08:00
Chris Lu
aa17311063 s3: set filename when uploading file chunks 2021-02-17 20:54:53 -08:00
Chris Lu
3f8b0da677 filer: do not print password on error 
fix https://github.com/chrislusf/seaweedfs/issues/1809
 2021-02-17 02:13:52 -08:00
Chris Lu
cd866664a8 skip JWT if fileId is empty 
related to https://github.com/chrislusf/seaweedfs/issues/1808
 2021-02-16 15:39:12 -08:00
Chris Lu
632c94d438 Merge branch 'extend_to_disk_type' 2021-02-16 11:06:13 -08:00
Chris Lu
0611233f16 adjust printing 2021-02-16 10:55:30 -08:00
Chris Lu
68775d29e3 fix tests 2021-02-16 10:51:03 -08:00
Chris Lu
7403cd43c5 only use "" for hdd to avoid two values for the same thing 2021-02-16 10:50:42 -08:00
Chris Lu
b314d78e97 fix print 2021-02-16 10:48:28 -08:00
Chris Lu
53ca7e66ef avoid dead lock 2021-02-16 10:48:16 -08:00
Chris Lu
38efc6f572 simplify 2021-02-16 10:47:32 -08:00
Chris Lu
ad70107eb4 ec balance among nodes with HDD 2021-02-16 06:22:50 -08:00
Chris Lu
3097b9a9b7 fix existence checking 2021-02-16 05:59:43 -08:00
Chris Lu
ebf320ec22 avoid divided by zero 2021-02-16 05:59:24 -08:00
Chris Lu
a0c6db361c avoid nil 2021-02-16 05:33:38 -08:00
Chris Lu
36f95e50a9 avoid possible nil disk info 2021-02-16 05:13:48 -08:00
Chris Lu
43101ccea0 move to the empty nodes first 2021-02-16 04:27:16 -08:00
Chris Lu
38bbef7ec1 avoid nil 2021-02-16 04:16:46 -08:00
Chris Lu
cb9cc29518 volume.list display; fix updating maxVolumeCount for disk 2021-02-16 03:55:24 -08:00
Chris Lu
3fe628f04e use hdd instead of empty string 2021-02-16 03:03:00 -08:00
Chris Lu
f8446b42ab this can compile now!!! 2021-02-16 02:47:02 -08:00
Chris Lu
b9b5b932c5 filer: postgres2 mysql2 avoid repeatedly creating tables 2021-02-15 16:19:24 -08:00
Chris Lu
71f0c19515 2.26 2021-02-15 13:38:38 -08:00
Chris Lu
8b169aa47c s3: Critical - S3 Delete Multiple Objects, delete the bucket 
fix https://github.com/chrislusf/seaweedfs/issues/1806
 2021-02-15 13:38:29 -08:00
Chris Lu
50202d1391
Merge pull request #1805 from LazyDBA247-Anyvision/master 
better postgresql connection pool management
 2021-02-14 22:26:29 -08:00
Chris Lu
8726ef34c1 2.25 2021-02-14 22:23:43 -08:00
LazyDBA247-Anyvision
7f458d5e78 better postgres connection pool management 
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime)
to enable refresh of stale connections.
 2021-02-15 07:45:09 +02:00
Chris Lu
0bc3a1f9e8 disk type only supports hdd and ssd, not ready for random tags yet 2021-02-14 11:38:43 -08:00
LazyDBA247-Anyvision
51b4963e2e postgres2 & memsql2 
add escape (quote identifiers) for the dynamic sql
so tables (collections) with special characters will work.
 2021-02-14 13:14:36 +02:00
Chris Lu
876dbe3d26 collect disk types 2021-02-13 23:25:16 -08:00
Chris Lu
861f8fab9b refactoring 2021-02-13 22:34:12 -08:00
Chris Lu
881f05f5a0 adjust help message 2021-02-13 16:13:52 -08:00
Chris Lu
ef76365ec2 adjust help message 2021-02-13 15:47:08 -08:00
Chris Lu
7ce647f27e support customizable disk type 2021-02-13 15:42:42 -08:00
Chris Lu
4bd8a692d8 disk type can be generic tags 2021-02-13 13:50:14 -08:00
Chris Lu
712b3e9e53 mount: release need to avoid nil reader 
fix https://github.com/chrislusf/seaweedfs/issues/1803
 2021-02-13 13:39:37 -08:00
Chris Lu
4ce56bac08 fix format error 2021-02-12 06:44:26 -08:00
Chris Lu
0604afcda7 mount: remove unnecessary flush on release 2021-02-12 06:01:04 -08:00
Chris Lu
7d9dc3c6a2 use fasthttp lib to read 2021-02-12 05:59:52 -08:00
Chris Lu
487e435679 adjust http max idle connections per host 
related to https://github.com/chrislusf/seaweedfs/issues/1802
 2021-02-12 03:47:15 -08:00
Chris Lu
0f426ce34d
Merge pull request #1801 from kmlebedev/recoveringRabbitMQ 
Do reconnect to RabbitMQ
 2021-02-11 01:05:03 -08:00
Konstantin Lebedev
94eac4f00e Do reconnect to RabbitMQ 
https://github.com/google/go-cloud/issues/2958
https://github.com/chrislusf/seaweedfs/issues/1773
 2021-02-11 13:59:36 +05:00