mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-11-25 03:29:10 +08:00
Code Issues Packages Projects Releases Wiki Activity
4,224 Commits 32 Branches 296 Tags 147 MiB
35f6518c36
Commit Graph

2608 Commits

Author SHA1 Message Date
Chris Lu
b8f32bcab9 filer: compress stored metadata 2020-09-03 11:00:20 -07:00
Chris Lu
f76a2b2c8a printout meta data size 2020-09-03 09:51:21 -07:00
ruitao.liu
798280e980 change filer2 to filer. 2020-09-03 17:05:26 +08:00
ruitao.liu
a93d27d1e8 new filer option to es v7. 2020-09-03 16:34:58 +08:00
Chris Lu
1d56ea24ef fix 2020-09-03 00:08:37 -07:00
Chris Lu
7e1aad0b54 mount: map uid/gid between local and filer 2020-09-03 00:07:22 -07:00
Chris Lu
68e878adb5 fix formatting 2020-09-02 21:42:12 -07:00
Chris Lu
0d99a5da3c c*: support kv operations 2020-09-02 18:39:24 -07:00
Chris Lu
645a4af3db mongodb: support kv operations 2020-09-02 17:19:14 -07:00
Chris Lu
ecaa30c408 better error message 2020-09-02 17:17:44 -07:00
Chris Lu
7c770b727c Update abstract_sql_store_kv.go 2020-09-02 10:13:48 -07:00
Chris Lu
05a6e2dc7f sql kv upsert 2020-09-02 10:12:44 -07:00
Chris Lu
ed62f52452 fix sql insert with duplicated primary key 2020-09-02 10:09:49 -07:00
Chris Lu
9ea290aa12 fix error message 2020-09-01 23:21:41 -07:00
Chris Lu
06a1188267 mysql/postgres: support kv operations 2020-09-01 22:48:36 -07:00
Chris Lu
aa40295f06 leveldb: report not found entry 2020-09-01 22:48:23 -07:00
Chris Lu
87d7312bf6 mysql/postgres: properly report entry not found 2020-09-01 22:47:57 -07:00
Chris Lu
72b0a5f1d1 mysql or postgres: log find error 2020-09-01 22:25:17 -07:00
Chris Lu
2815bbe6c0 go fmt 2020-09-01 21:59:26 -07:00
Chris Lu
37234bf3f8 filer store adds kv support 
can compile now, need to implement those unimplemented
 2020-09-01 21:59:00 -07:00
James Hartig
8e54e34576 volume: Don't unmount before deleting volume in copy 
If we unmount first and then delete, the delete fails because the volume
was unmounted. Delete ends up doing the same thing as the unmount anyways.
 2020-09-01 22:00:07 -04:00
Chris Lu
2b14ae5819 fix tests 2020-09-01 01:33:43 -07:00
Chris Lu
d91ec535b3 fix tests 2020-09-01 01:29:13 -07:00
Chris Lu
eb7929a971 rename filer2 to filer 2020-09-01 00:21:19 -07:00
Chris Lu
38e06d783d volume: check disk space before compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1440
 2020-08-31 18:10:53 -07:00
Chris Lu
dbb10e0f0b testing with more than 30days 2020-08-31 17:59:29 -07:00
Chris Lu
dd176cdb8b avoid tiny step pagination 2020-08-31 11:28:03 -07:00
Chris Lu
97733c3dff typo 2020-08-31 10:49:17 -07:00
Chris Lu
c8398bdb09 adjust metrics 2020-08-31 10:41:05 -07:00
Chris Lu
8a1c8e41b3 simplify if else logic 2020-08-31 10:39:24 -07:00
Chris Lu
f0c89cfacd go fmt 2020-08-31 10:23:31 -07:00
Chris Lu
edb9d65e05
Merge pull request #1431 from kmlebedev/wip-prefix-search 
Wip prefix search
 2020-08-31 10:22:14 -07:00
Konstantin Lebedev
9a195bebfd accurate limit 2020-08-31 22:13:13 +05:00
Konstantin Lebedev
18b98f9747 not convert err to string 2020-08-31 21:55:18 +05:00
Konstantin Lebedev
22f32e75c4 rename 2020-08-31 21:52:05 +05:00
Konstantin Lebedev
a8de332beb fix 2020-08-31 21:45:44 +05:00
Konstantin Lebedev
7f013d77d6 add var UnsupportedListDirectoryPrefixedErr 2020-08-31 21:43:32 +05:00
Konstantin Lebedev
60a86cfe04 add const 2020-08-31 21:35:16 +05:00
Konstantin Lebedev
82ea121d09 rm func 2020-08-31 18:50:06 +05:00
Konstantin Lebedev
2ea638f865 filer store wrapper can implement the logic to filter by prefi 2020-08-31 17:13:56 +05:00
Chris Lu
408e339c53 also delete the manifest chunk itself 2020-08-31 00:16:03 -07:00
Chris Lu
dd52037c27 1.92 2020-08-30 22:56:35 -07:00
Chris Lu
33d8c6c617 change log to warning 2020-08-30 21:01:44 -07:00
Chris Lu
e62d7f221f a bit more logs 2020-08-30 21:00:24 -07:00
Chris Lu
22fe4ae573 mount: fix for UrBackup 2020-08-30 21:00:09 -07:00
Chris Lu
ae84a9ee2e add logs 2020-08-30 20:59:04 -07:00
Chris Lu
a41588279a change log level 5 to 4 2020-08-30 20:12:04 -07:00
Chris Lu
56244fb9a1 fix hard coded host address 2020-08-30 10:23:35 -07:00
Chris Lu
f2a8574448 filer and mount deletion resolves manifest chunks also 2020-08-30 02:07:14 -07:00
Chris Lu
99ecf63276 go fmt 2020-08-29 22:28:33 -07:00
Chris Lu
ef75ce8a34 use UTC for all time related folders 2020-08-29 21:01:14 -07:00
Chris Lu
b69cb74c03 read meta logs by timestamp 
pass in event ts when moving logs
meta aggregator reads in memory logs only
 2020-08-29 17:37:19 -07:00
Chris Lu
063c9ddac5 adjust logs 2020-08-29 11:56:22 -07:00
Chris Lu
ca658a97c5 add signatures to messages to avoid double processing 2020-08-28 23:48:48 -07:00
Chris Lu
63ad1abcce watch: follow changes for a pattern 2020-08-28 19:43:04 -07:00
Chris Lu
b7a654a318 file mime avoid saving application/octet-stream 2020-08-28 19:42:40 -07:00
Chris Lu
707192f966 fix help message 2020-08-27 00:13:38 -07:00
Chris Lu
0be6863c87 rename 2020-08-26 22:40:15 -07:00
Chris Lu
50f7e2e89f add log messages 2020-08-26 09:51:20 -07:00
Chris Lu
6bc618051c log message 2020-08-26 09:32:56 -07:00
Chris Lu
9dae4e9d3e fix logs 2020-08-26 09:31:11 -07:00
Chris Lu
4c1a3187bf fix log error 2020-08-26 09:25:54 -07:00
Chris Lu
ab759f0ec2 erasure coding: fix EC error if multiple disks are configured in one volume server 2020-08-26 09:16:58 -07:00
Chris Lu
d1cf39f180 fix logging 2020-08-26 08:55:15 -07:00
Chris Lu
83012f543a s: avoid possible completeMultipartUpload re-tries that can overwrite the uploaded file 2020-08-26 08:50:24 -07:00
Konstantin Lebedev
464d4c82ec stop send heartbeat before stop volume server 2020-08-26 16:16:11 +05:00
Chris Lu
1901f15cd2 volume server: remove whitelist for status checking 
Volume liveness/readiness not work if jwt enabled, I try use param
-whiteList
but jwt breaks
 2020-08-25 02:27:47 -07:00
Chris Lu
81e5124faf s3: list bucket permission change from admin to read 
fix https://github.com/chrislusf/seaweedfs/issues/1430
 2020-08-24 11:22:45 -07:00
Chris Lu
5f55a87101 close http response 2020-08-24 00:32:44 -07:00
Chris Lu
0b301b0b47 consume all response body in order to release requests 2020-08-23 21:32:29 -07:00
Chris Lu
2cdad77d82 fix checkin error 2020-08-23 18:30:11 -07:00
Chris Lu
81a8aa6581 1.91 2020-08-23 17:28:40 -07:00
Chris Lu
51346a5930 always keep the manifest list of chunks 2020-08-23 17:15:12 -07:00
Chris Lu
c1d1677a28 keep manifest chunks forever 2020-08-23 16:59:01 -07:00
Chris Lu
98175548c2 adjust deletion ordering 2020-08-23 16:27:12 -07:00
Chris Lu
f0e50e7b26 fix test 2020-08-23 16:12:34 -07:00
Chris Lu
aee27ccbe1 multiple fixes 
* adjust isOpen count
* move ContinuousDirtyPages lock to filehandle
* fix problem with MergeIntoVisibles, avoid reusing slices
* let filer delete the garbage
 2020-08-23 15:48:02 -07:00
Chris Lu
c19245886c simpler test 2020-08-23 14:20:27 -07:00
Chris Lu
df816a58fe add tests 2020-08-23 14:09:25 -07:00
Chris Lu
77393d3d30 add file handle locking when changing file entry 2020-08-23 00:35:50 -07:00
Chris Lu
d60bcbf08a sorting chunks 2020-08-23 00:00:36 -07:00
Chris Lu
5e6b714836 add random test 2020-08-22 16:33:00 -07:00
Chris Lu
42ddbcc3a2 SQL insert falls back to update 
fix https://github.com/chrislusf/seaweedfs/issues/1429
 2020-08-20 07:52:46 -07:00
Chris Lu
f48567c5c6 remove unused function 2020-08-19 22:53:49 -07:00
Chris Lu
c45ba5d7d4 fix listObjectsV2 response format 
fix https://github.com/chrislusf/seaweedfs/issues/1426 issue 1
 2020-08-19 12:08:04 -07:00
James Hartig
3ccfa4c6ad Added VolumeMarkWritable and VolumeStatus grpc methods 
This is necessary for copy to mark as read-only and then restore the
original state afterwards.
 2020-08-19 11:42:56 -04:00
Chris Lu
3b4b1d4a77 fix tests 2020-08-19 01:37:56 -07:00
Chris Lu
839634097f also do flush on release 2020-08-19 01:27:10 -07:00
Chris Lu
c27e18aa6a read possible old deleted chunks 2020-08-19 00:43:07 -07:00
Chris Lu
a78772d5ea avoid shutdown in the middle of running 2020-08-19 00:42:02 -07:00
Chris Lu
ae9bc4a508 logs 2020-08-18 23:42:26 -07:00
Chris Lu
ed4b43b419 adjust logs 2020-08-18 23:42:09 -07:00
Chris Lu
9f1e0aeef5 delete chunks in the last step 2020-08-18 23:40:53 -07:00
Chris Lu
99d05f758c adjust logs 2020-08-18 23:39:18 -07:00
Chris Lu
fe01191b5b support read option readDeleted=true 2020-08-18 19:22:16 -07:00
Chris Lu
6ccd7f0a4d refactoring 2020-08-18 18:01:37 -07:00
Chris Lu
c026eb0592 refactoring 2020-08-18 17:39:29 -07:00
Chris Lu
7e91ae592c pass in option to read deleted entries 
not working yet
 2020-08-18 17:37:26 -07:00
Chris Lu
51ecb49db3 for debugging 2020-08-18 17:35:39 -07:00
Chris Lu
ee11d98650 refactoring 2020-08-18 17:35:19 -07:00
Chris Lu
6a92f0bc7a refactoring to typed Size 
Go is amazing with refactoring!
 2020-08-18 17:04:28 -07:00
Chris Lu
618b2f6829 release resources only when needed to 2020-08-18 12:53:08 -07:00
Chris Lu
208849702d logs 2020-08-18 12:52:54 -07:00
Chris Lu
1fcd083db3 printout data size 2020-08-18 09:09:29 -07:00
Chris Lu
3e5339337a minor 2020-08-18 08:50:14 -07:00
Chris Lu
85001cbec7 properly report io.EOF 2020-08-18 08:18:54 -07:00
Chris Lu
0ca45a5cbc adjust logs 2020-08-18 00:34:15 -07:00
Chris Lu
cd43738245 fix reading when filling zeros 2020-08-18 00:32:01 -07:00
Chris Lu
ecb3ce46be adjust error logs 2020-08-18 00:31:42 -07:00
Chris Lu
30fe424469 add one more test case 2020-08-17 22:47:27 -07:00
Chris Lu
1b68ba953b fix for out of range reads 2020-08-17 22:46:32 -07:00
Chris Lu
56fbd2c211 fix reading 2020-08-17 21:17:32 -07:00
Chris Lu
09e126bae5 refactoring: use interface 2020-08-17 20:20:08 -07:00
Chris Lu
be4d42b8e2 rename 2020-08-17 20:15:53 -07:00
Chris Lu
0625e63648 count 0 as part of the reads 2020-08-17 20:14:40 -07:00
Chris Lu
0be4b6e7f0 logs 2020-08-17 16:05:40 -07:00
Chris Lu
97e54a80d4 rename variables 2020-08-17 16:05:13 -07:00
Chris Lu
abdaf9958d possibly read more 2020-08-17 16:04:56 -07:00
Chris Lu
9d46c7bc78 rename 2020-08-17 11:12:10 -07:00
Chris Lu
4ccfdaeb4d prevent nil 2020-08-17 10:07:34 -07:00
Chris Lu
24c8e6bcb4 minor optimization 2020-08-17 10:03:34 -07:00
Chris Lu
f5837b7000 report error first 2020-08-16 23:49:10 -07:00
Chris Lu
2ac27616bc fix possible out of range bytes 
avoid buff out of range resp.Data = buff[:totalRead]
 2020-08-16 23:47:34 -07:00
Chris Lu
ff200398bb 1.90 2020-08-16 21:18:50 -07:00
Chris Lu
4ceeba9e70 streaming reads 2020-08-16 21:07:46 -07:00
Chris Lu
e72953dff7 logs 2020-08-16 21:06:03 -07:00
Chris Lu
627b081b67 adjust logs 2020-08-16 16:32:22 -07:00
Chris Lu
22e5132b3a adjust log level 2020-08-16 16:25:11 -07:00
Chris Lu
ee0f92a6be reduce memory allocation 2020-08-16 16:24:40 -07:00
Chris Lu
2ba817afac read randomly written data 2020-08-16 15:16:46 -07:00
Chris Lu
6111b265e7 fix compilation 2020-08-16 01:38:16 -07:00
Chris Lu
4a77f0820a clean up logs 2020-08-16 01:37:50 -07:00
Chris Lu
8c9e6eaacd fix tests 2020-08-16 01:35:52 -07:00
Chris Lu
b71df82292 1.89 2020-08-16 00:57:28 -07:00
Chris Lu
1d9ea30b72 fix ViewFromVisibleIntervals 2020-08-16 00:49:26 -07:00
Chris Lu
aec7f32b02 fix reader_at 2020-08-16 00:49:08 -07:00
Chris Lu
5d80fc2ec7 adjust logs 2020-08-15 21:09:31 -07:00
Chris Lu
e0bfd3161a update metadata only if changed 2020-08-15 19:56:24 -07:00
Chris Lu
003d48da21 adjust logs 2020-08-15 19:55:28 -07:00
Chris Lu
6ee8d952d2 adjust log level 2020-08-15 18:24:35 -07:00
Chris Lu
3e1395b767 adjust log message 2020-08-15 17:06:16 -07:00
Chris Lu
a22ee30596 fix nil 2020-08-15 17:01:42 -07:00
Chris Lu
0d60e67816 ensure meta data changes are updated 2020-08-15 14:15:07 -07:00
Chris Lu
5b43bddf20 proper deletion ordering 
delete central file store first, then delete local cache
 2020-08-15 09:33:41 -07:00
Chris Lu
c647deace1 file size support set file length 
use Attr.FileSize and TotalChunkSize to determine file size
 2020-08-15 09:32:47 -07:00
Chris Lu
c03bb180eb fix error reporting 2020-08-14 00:44:02 -07:00
Chris Lu
eb493283dd deletion error report 2020-08-14 00:23:01 -07:00
Chris Lu
a7f669044e rename also applies to open file handle 2020-08-14 00:22:49 -07:00
Chris Lu
edfa73782f adjust log level 2020-08-14 00:22:21 -07:00
Chris Lu
0983060a90 increase default volume file size limit to 1024 
avoid possible large chunk size set on mount or filer
 2020-08-13 09:07:22 -07:00
Chris Lu
090612492c fix compilation 2020-08-13 00:07:56 -07:00
Chris Lu
d43129d27f add back handles lock 2020-08-12 23:52:13 -07:00
Chris Lu
f735d579d3 adjust comment 2020-08-12 13:22:19 -07:00
Chris Lu
8824a9755c remove directory cache 2020-08-12 13:11:04 -07:00
Chris Lu
83cad3da79 add retry file upload 2020-08-11 20:30:11 -07:00
Chris Lu
152a6cbc2b minor adjustments 2020-08-10 20:42:27 -07:00
Chris Lu
f1b40567e3
Merge pull request #1419 from robin1900/master 
fix bug:  two same volumeId in different collections
 2020-08-10 20:38:55 -07:00
cheng.li01
25fbff5d52 fix bug: two same volumeId in different collections 
1, there will be two leader when master server startup in a few seconds
2, raft server will get a leader even there is only one master, so there is no need to do hard code to set the server to be leader
 2020-08-10 16:37:47 +08:00
Chris Lu
e74dc4e4bc add back fs node cache for renaming 2020-08-09 21:56:09 -07:00
Chris Lu
f86c7d911a remove println 2020-08-09 14:42:25 -07:00
Chris Lu
9ecc1170a3 istObjects和listObjectsV2不能查询子目录 #1418 
fix https://github.com/chrislusf/seaweedfs/issues/1418
 2020-08-09 14:35:53 -07:00
Chris Lu
3f4aff5dde s3: fix delimiter in list response 2020-08-09 09:09:35 -07:00
Chris Lu
b056707770 1.88 2020-08-08 22:55:12 -07:00
Chris Lu
9832653e1d FUSE mount: proper error with deleting non empty folder 2020-08-08 21:37:36 -07:00
Chris Lu
3b1a95ac26 filer refactoring: same auto chunking logic for POST and PUT, no size limit 2020-08-08 12:02:06 -07:00
Chris Lu
67348e7b15 less noisy heartbeat logs 2020-08-08 10:53:35 -07:00
Chris Lu
bee0d7e5eb lower log priority for noisy heartbeat 2020-08-08 10:52:13 -07:00
Chris Lu
ae00cce4bd support POST and PUT auto chunking 2020-08-08 10:45:37 -07:00
Chris Lu
bd8bfdae07 refactoring 2020-08-08 10:18:43 -07:00
Chris Lu
ab6e5c0dc4 adjust error message 2020-08-08 10:18:32 -07:00
Chris Lu
828a5ae429 check signature only when auth is enabled 2020-08-08 09:11:40 -07:00
Chris Lu
20e2ac1add filer: store md5 metadata for files uploaded by filer 
fix https://github.com/chrislusf/seaweedfs/issues/1412
 2020-08-06 10:04:17 -07:00
Chris Lu
93ea0801ea volume: the variable for the master node may be stale? 
related to https://github.com/chrislusf/seaweedfs/issues/1414
 2020-08-06 09:48:54 -07:00
Chris Lu
4ecfa9879d volume: report Content-MD5 in response header 2020-08-06 05:22:53 -07:00
Chris Lu
41007ced77 remove logging 2020-08-06 04:26:29 -07:00
Chris Lu
2b74abf766 S3: configurable access for anonymous user 
fix https://github.com/chrislusf/seaweedfs/issues/1413
 2020-08-06 03:41:34 -07:00
Устюжанин Антон Александрович
9ca011e3ca ListDirectoryPrefixedEntries 2020-08-06 00:38:59 +05:00
Устюжанин Антон Александрович
b231f7bdab ListDirectoryPrefixedEntries 2020-08-06 00:37:42 +05:00
Устюжанин Антон Александрович
a457c308ad test ListDirectoryPrefixedEntries 2020-08-06 00:24:31 +05:00
Устюжанин Антон Александрович
2d3b355fb6 test ListDirectoryPrefixedEntries 2020-08-05 23:56:06 +05:00
Устюжанин Антон Александрович
67f134ebd7 test ListDirectoryPrefixedEntries 2020-08-05 23:44:48 +05:00
Устюжанин Антон Александрович
dc9fc01254 test ListDirectoryPrefixedEntries 2020-08-05 23:38:00 +05:00
Устюжанин Антон Александрович
33a9e5e2d1 test ListDirectoryPrefixedEntries 2020-08-05 22:19:16 +05:00
Chris Lu
cbd80253e3 better needle id format 2020-08-04 22:09:07 -07:00
Chris Lu
1b3a80dd3d non-fatal error 2020-08-01 13:46:52 -07:00
Chris Lu
bbbea8159c http request use gzip if possible 2020-08-01 13:20:52 -07:00
Chris Lu
2f03481cb2 in case when content is not compressed 2020-08-01 13:08:10 -07:00
Chris Lu
49199c69d4 volume: avoid special logic for .gz files 2020-08-01 11:19:57 -07:00
Chris Lu
f89186a3cf better logging 2020-08-01 11:19:57 -07:00
Chris Lu
49d6dcfbc6 dynamically change the log level for reading the disk status 2020-08-01 11:19:57 -07:00
limd
b41b7ea4d0 fix spark read s3 bug (sc.binaryFiles) 2020-08-01 01:08:30 +08:00
Chris Lu
a6b59d50f7 1.87 2020-07-30 13:52:04 -07:00
Chris Lu
eed525b717 FUSE mount: remove DirListCacheLimit 
outdated parameter
 2020-07-28 09:24:39 -07:00
Chris Lu
a566bfc6e1 s3: use bucket in the domain 
fix https://github.com/chrislusf/seaweedfs/issues/1405
 2020-07-28 08:47:27 -07:00
limd
92c32f9d46 PutObject: fix create folder bug 2020-07-28 00:58:42 +08:00
Chris Lu
54cd582653 refactor 2020-07-26 12:58:58 -07:00
Chris Lu
c50ec1599c auth for presigned put object part 2020-07-26 12:09:50 -07:00
Chris Lu
ea0180de4e auth for presigned put operation 2020-07-26 12:06:45 -07:00
Chris Lu
3faa0aa448 rename 2020-07-25 20:09:19 -07:00
Chris Lu
023a1efdf2 set filename in Content-Disposition header 2020-07-25 20:06:40 -07:00