mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-11-30 15:19:01 +08:00
Code Issues Packages Projects Releases Wiki Activity
4,357 Commits 32 Branches 296 Tags 148 MiB
69f0da5d73
Commit Graph

2698 Commits

Author SHA1 Message Date
Chris Lu
7c770b727c Update abstract_sql_store_kv.go 2020-09-02 10:13:48 -07:00
Chris Lu
05a6e2dc7f sql kv upsert 2020-09-02 10:12:44 -07:00
Chris Lu
ed62f52452 fix sql insert with duplicated primary key 2020-09-02 10:09:49 -07:00
Chris Lu
9ea290aa12 fix error message 2020-09-01 23:21:41 -07:00
Chris Lu
06a1188267 mysql/postgres: support kv operations 2020-09-01 22:48:36 -07:00
Chris Lu
aa40295f06 leveldb: report not found entry 2020-09-01 22:48:23 -07:00
Chris Lu
87d7312bf6 mysql/postgres: properly report entry not found 2020-09-01 22:47:57 -07:00
Chris Lu
72b0a5f1d1 mysql or postgres: log find error 2020-09-01 22:25:17 -07:00
Chris Lu
2815bbe6c0 go fmt 2020-09-01 21:59:26 -07:00
Chris Lu
37234bf3f8 filer store adds kv support 
can compile now, need to implement those unimplemented
 2020-09-01 21:59:00 -07:00
James Hartig
8e54e34576 volume: Don't unmount before deleting volume in copy 
If we unmount first and then delete, the delete fails because the volume
was unmounted. Delete ends up doing the same thing as the unmount anyways.
 2020-09-01 22:00:07 -04:00
Chris Lu
2b14ae5819 fix tests 2020-09-01 01:33:43 -07:00
Chris Lu
d91ec535b3 fix tests 2020-09-01 01:29:13 -07:00
Chris Lu
eb7929a971 rename filer2 to filer 2020-09-01 00:21:19 -07:00
Chris Lu
38e06d783d volume: check disk space before compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1440
 2020-08-31 18:10:53 -07:00
Chris Lu
dbb10e0f0b testing with more than 30days 2020-08-31 17:59:29 -07:00
Chris Lu
dd176cdb8b avoid tiny step pagination 2020-08-31 11:28:03 -07:00
Chris Lu
97733c3dff typo 2020-08-31 10:49:17 -07:00
Chris Lu
c8398bdb09 adjust metrics 2020-08-31 10:41:05 -07:00
Chris Lu
8a1c8e41b3 simplify if else logic 2020-08-31 10:39:24 -07:00
Chris Lu
f0c89cfacd go fmt 2020-08-31 10:23:31 -07:00
Chris Lu
edb9d65e05
Merge pull request #1431 from kmlebedev/wip-prefix-search 
Wip prefix search
 2020-08-31 10:22:14 -07:00
Konstantin Lebedev
9a195bebfd accurate limit 2020-08-31 22:13:13 +05:00
Konstantin Lebedev
18b98f9747 not convert err to string 2020-08-31 21:55:18 +05:00
Konstantin Lebedev
22f32e75c4 rename 2020-08-31 21:52:05 +05:00
Konstantin Lebedev
a8de332beb fix 2020-08-31 21:45:44 +05:00
Konstantin Lebedev
7f013d77d6 add var UnsupportedListDirectoryPrefixedErr 2020-08-31 21:43:32 +05:00
Konstantin Lebedev
60a86cfe04 add const 2020-08-31 21:35:16 +05:00
Konstantin Lebedev
82ea121d09 rm func 2020-08-31 18:50:06 +05:00
Konstantin Lebedev
2ea638f865 filer store wrapper can implement the logic to filter by prefi 2020-08-31 17:13:56 +05:00
Chris Lu
408e339c53 also delete the manifest chunk itself 2020-08-31 00:16:03 -07:00
Chris Lu
dd52037c27 1.92 2020-08-30 22:56:35 -07:00
Chris Lu
33d8c6c617 change log to warning 2020-08-30 21:01:44 -07:00
Chris Lu
e62d7f221f a bit more logs 2020-08-30 21:00:24 -07:00
Chris Lu
22fe4ae573 mount: fix for UrBackup 2020-08-30 21:00:09 -07:00
Chris Lu
ae84a9ee2e add logs 2020-08-30 20:59:04 -07:00
Chris Lu
a41588279a change log level 5 to 4 2020-08-30 20:12:04 -07:00
Chris Lu
56244fb9a1 fix hard coded host address 2020-08-30 10:23:35 -07:00
Chris Lu
f2a8574448 filer and mount deletion resolves manifest chunks also 2020-08-30 02:07:14 -07:00
Chris Lu
99ecf63276 go fmt 2020-08-29 22:28:33 -07:00
Chris Lu
ef75ce8a34 use UTC for all time related folders 2020-08-29 21:01:14 -07:00
Chris Lu
b69cb74c03 read meta logs by timestamp 
pass in event ts when moving logs
meta aggregator reads in memory logs only
 2020-08-29 17:37:19 -07:00
Chris Lu
063c9ddac5 adjust logs 2020-08-29 11:56:22 -07:00
Chris Lu
ca658a97c5 add signatures to messages to avoid double processing 2020-08-28 23:48:48 -07:00
Chris Lu
63ad1abcce watch: follow changes for a pattern 2020-08-28 19:43:04 -07:00
Chris Lu
b7a654a318 file mime avoid saving application/octet-stream 2020-08-28 19:42:40 -07:00
Chris Lu
707192f966 fix help message 2020-08-27 00:13:38 -07:00
Chris Lu
0be6863c87 rename 2020-08-26 22:40:15 -07:00
Chris Lu
50f7e2e89f add log messages 2020-08-26 09:51:20 -07:00
Chris Lu
6bc618051c log message 2020-08-26 09:32:56 -07:00
Chris Lu
9dae4e9d3e fix logs 2020-08-26 09:31:11 -07:00
Chris Lu
4c1a3187bf fix log error 2020-08-26 09:25:54 -07:00
Chris Lu
ab759f0ec2 erasure coding: fix EC error if multiple disks are configured in one volume server 2020-08-26 09:16:58 -07:00
Chris Lu
d1cf39f180 fix logging 2020-08-26 08:55:15 -07:00
Chris Lu
83012f543a s: avoid possible completeMultipartUpload re-tries that can overwrite the uploaded file 2020-08-26 08:50:24 -07:00
Konstantin Lebedev
464d4c82ec stop send heartbeat before stop volume server 2020-08-26 16:16:11 +05:00
Chris Lu
1901f15cd2 volume server: remove whitelist for status checking 
Volume liveness/readiness not work if jwt enabled, I try use param
-whiteList
but jwt breaks
 2020-08-25 02:27:47 -07:00
Chris Lu
81e5124faf s3: list bucket permission change from admin to read 
fix https://github.com/chrislusf/seaweedfs/issues/1430
 2020-08-24 11:22:45 -07:00
Chris Lu
5f55a87101 close http response 2020-08-24 00:32:44 -07:00
Chris Lu
0b301b0b47 consume all response body in order to release requests 2020-08-23 21:32:29 -07:00
Chris Lu
2cdad77d82 fix checkin error 2020-08-23 18:30:11 -07:00
Chris Lu
81a8aa6581 1.91 2020-08-23 17:28:40 -07:00
Chris Lu
51346a5930 always keep the manifest list of chunks 2020-08-23 17:15:12 -07:00
Chris Lu
c1d1677a28 keep manifest chunks forever 2020-08-23 16:59:01 -07:00
Chris Lu
98175548c2 adjust deletion ordering 2020-08-23 16:27:12 -07:00
Chris Lu
f0e50e7b26 fix test 2020-08-23 16:12:34 -07:00
Chris Lu
aee27ccbe1 multiple fixes 
* adjust isOpen count
* move ContinuousDirtyPages lock to filehandle
* fix problem with MergeIntoVisibles, avoid reusing slices
* let filer delete the garbage
 2020-08-23 15:48:02 -07:00
Chris Lu
c19245886c simpler test 2020-08-23 14:20:27 -07:00
Chris Lu
df816a58fe add tests 2020-08-23 14:09:25 -07:00
Chris Lu
77393d3d30 add file handle locking when changing file entry 2020-08-23 00:35:50 -07:00
Chris Lu
d60bcbf08a sorting chunks 2020-08-23 00:00:36 -07:00
Chris Lu
5e6b714836 add random test 2020-08-22 16:33:00 -07:00
Chris Lu
42ddbcc3a2 SQL insert falls back to update 
fix https://github.com/chrislusf/seaweedfs/issues/1429
 2020-08-20 07:52:46 -07:00
Chris Lu
f48567c5c6 remove unused function 2020-08-19 22:53:49 -07:00
Chris Lu
c45ba5d7d4 fix listObjectsV2 response format 
fix https://github.com/chrislusf/seaweedfs/issues/1426 issue 1
 2020-08-19 12:08:04 -07:00
James Hartig
3ccfa4c6ad Added VolumeMarkWritable and VolumeStatus grpc methods 
This is necessary for copy to mark as read-only and then restore the
original state afterwards.
 2020-08-19 11:42:56 -04:00
Chris Lu
3b4b1d4a77 fix tests 2020-08-19 01:37:56 -07:00
Chris Lu
839634097f also do flush on release 2020-08-19 01:27:10 -07:00
Chris Lu
c27e18aa6a read possible old deleted chunks 2020-08-19 00:43:07 -07:00
Chris Lu
a78772d5ea avoid shutdown in the middle of running 2020-08-19 00:42:02 -07:00
Chris Lu
ae9bc4a508 logs 2020-08-18 23:42:26 -07:00
Chris Lu
ed4b43b419 adjust logs 2020-08-18 23:42:09 -07:00
Chris Lu
9f1e0aeef5 delete chunks in the last step 2020-08-18 23:40:53 -07:00
Chris Lu
99d05f758c adjust logs 2020-08-18 23:39:18 -07:00
Chris Lu
fe01191b5b support read option readDeleted=true 2020-08-18 19:22:16 -07:00
Chris Lu
6ccd7f0a4d refactoring 2020-08-18 18:01:37 -07:00
Chris Lu
c026eb0592 refactoring 2020-08-18 17:39:29 -07:00
Chris Lu
7e91ae592c pass in option to read deleted entries 
not working yet
 2020-08-18 17:37:26 -07:00
Chris Lu
51ecb49db3 for debugging 2020-08-18 17:35:39 -07:00
Chris Lu
ee11d98650 refactoring 2020-08-18 17:35:19 -07:00
Chris Lu
6a92f0bc7a refactoring to typed Size 
Go is amazing with refactoring!
 2020-08-18 17:04:28 -07:00
Chris Lu
618b2f6829 release resources only when needed to 2020-08-18 12:53:08 -07:00
Chris Lu
208849702d logs 2020-08-18 12:52:54 -07:00
Chris Lu
1fcd083db3 printout data size 2020-08-18 09:09:29 -07:00
Chris Lu
3e5339337a minor 2020-08-18 08:50:14 -07:00
Chris Lu
85001cbec7 properly report io.EOF 2020-08-18 08:18:54 -07:00
Chris Lu
0ca45a5cbc adjust logs 2020-08-18 00:34:15 -07:00
Chris Lu
cd43738245 fix reading when filling zeros 2020-08-18 00:32:01 -07:00
Chris Lu
ecb3ce46be adjust error logs 2020-08-18 00:31:42 -07:00
Chris Lu
30fe424469 add one more test case 2020-08-17 22:47:27 -07:00
Chris Lu
1b68ba953b fix for out of range reads 2020-08-17 22:46:32 -07:00
Chris Lu
56fbd2c211 fix reading 2020-08-17 21:17:32 -07:00
Chris Lu
09e126bae5 refactoring: use interface 2020-08-17 20:20:08 -07:00
Chris Lu
be4d42b8e2 rename 2020-08-17 20:15:53 -07:00
Chris Lu
0625e63648 count 0 as part of the reads 2020-08-17 20:14:40 -07:00
Chris Lu
0be4b6e7f0 logs 2020-08-17 16:05:40 -07:00
Chris Lu
97e54a80d4 rename variables 2020-08-17 16:05:13 -07:00
Chris Lu
abdaf9958d possibly read more 2020-08-17 16:04:56 -07:00
Chris Lu
9d46c7bc78 rename 2020-08-17 11:12:10 -07:00
Chris Lu
4ccfdaeb4d prevent nil 2020-08-17 10:07:34 -07:00
Chris Lu
24c8e6bcb4 minor optimization 2020-08-17 10:03:34 -07:00
Chris Lu
f5837b7000 report error first 2020-08-16 23:49:10 -07:00
Chris Lu
2ac27616bc fix possible out of range bytes 
avoid buff out of range resp.Data = buff[:totalRead]
 2020-08-16 23:47:34 -07:00
Chris Lu
ff200398bb 1.90 2020-08-16 21:18:50 -07:00
Chris Lu
4ceeba9e70 streaming reads 2020-08-16 21:07:46 -07:00
Chris Lu
e72953dff7 logs 2020-08-16 21:06:03 -07:00
Chris Lu
627b081b67 adjust logs 2020-08-16 16:32:22 -07:00
Chris Lu
22e5132b3a adjust log level 2020-08-16 16:25:11 -07:00
Chris Lu
ee0f92a6be reduce memory allocation 2020-08-16 16:24:40 -07:00
Chris Lu
2ba817afac read randomly written data 2020-08-16 15:16:46 -07:00
Chris Lu
6111b265e7 fix compilation 2020-08-16 01:38:16 -07:00
Chris Lu
4a77f0820a clean up logs 2020-08-16 01:37:50 -07:00
Chris Lu
8c9e6eaacd fix tests 2020-08-16 01:35:52 -07:00
Chris Lu
b71df82292 1.89 2020-08-16 00:57:28 -07:00
Chris Lu
1d9ea30b72 fix ViewFromVisibleIntervals 2020-08-16 00:49:26 -07:00
Chris Lu
aec7f32b02 fix reader_at 2020-08-16 00:49:08 -07:00
Chris Lu
5d80fc2ec7 adjust logs 2020-08-15 21:09:31 -07:00
Chris Lu
e0bfd3161a update metadata only if changed 2020-08-15 19:56:24 -07:00
Chris Lu
003d48da21 adjust logs 2020-08-15 19:55:28 -07:00
Chris Lu
6ee8d952d2 adjust log level 2020-08-15 18:24:35 -07:00
Chris Lu
3e1395b767 adjust log message 2020-08-15 17:06:16 -07:00
Chris Lu
a22ee30596 fix nil 2020-08-15 17:01:42 -07:00
Chris Lu
0d60e67816 ensure meta data changes are updated 2020-08-15 14:15:07 -07:00
Chris Lu
5b43bddf20 proper deletion ordering 
delete central file store first, then delete local cache
 2020-08-15 09:33:41 -07:00
Chris Lu
c647deace1 file size support set file length 
use Attr.FileSize and TotalChunkSize to determine file size
 2020-08-15 09:32:47 -07:00
Chris Lu
c03bb180eb fix error reporting 2020-08-14 00:44:02 -07:00
Chris Lu
eb493283dd deletion error report 2020-08-14 00:23:01 -07:00
Chris Lu
a7f669044e rename also applies to open file handle 2020-08-14 00:22:49 -07:00
Chris Lu
edfa73782f adjust log level 2020-08-14 00:22:21 -07:00
Chris Lu
0983060a90 increase default volume file size limit to 1024 
avoid possible large chunk size set on mount or filer
 2020-08-13 09:07:22 -07:00
Chris Lu
090612492c fix compilation 2020-08-13 00:07:56 -07:00
Chris Lu
d43129d27f add back handles lock 2020-08-12 23:52:13 -07:00
Chris Lu
f735d579d3 adjust comment 2020-08-12 13:22:19 -07:00
Chris Lu
8824a9755c remove directory cache 2020-08-12 13:11:04 -07:00
Chris Lu
83cad3da79 add retry file upload 2020-08-11 20:30:11 -07:00
Chris Lu
152a6cbc2b minor adjustments 2020-08-10 20:42:27 -07:00
Chris Lu
f1b40567e3
Merge pull request #1419 from robin1900/master 
fix bug:  two same volumeId in different collections
 2020-08-10 20:38:55 -07:00
cheng.li01
25fbff5d52 fix bug: two same volumeId in different collections 
1, there will be two leader when master server startup in a few seconds
2, raft server will get a leader even there is only one master, so there is no need to do hard code to set the server to be leader
 2020-08-10 16:37:47 +08:00
Chris Lu
e74dc4e4bc add back fs node cache for renaming 2020-08-09 21:56:09 -07:00
Chris Lu
f86c7d911a remove println 2020-08-09 14:42:25 -07:00
Chris Lu
9ecc1170a3 istObjects和listObjectsV2不能查询子目录 #1418 
fix https://github.com/chrislusf/seaweedfs/issues/1418
 2020-08-09 14:35:53 -07:00
Chris Lu
3f4aff5dde s3: fix delimiter in list response 2020-08-09 09:09:35 -07:00
Chris Lu
b056707770 1.88 2020-08-08 22:55:12 -07:00
Chris Lu
9832653e1d FUSE mount: proper error with deleting non empty folder 2020-08-08 21:37:36 -07:00
Chris Lu
3b1a95ac26 filer refactoring: same auto chunking logic for POST and PUT, no size limit 2020-08-08 12:02:06 -07:00
Chris Lu
67348e7b15 less noisy heartbeat logs 2020-08-08 10:53:35 -07:00
Chris Lu
bee0d7e5eb lower log priority for noisy heartbeat 2020-08-08 10:52:13 -07:00
Chris Lu
ae00cce4bd support POST and PUT auto chunking 2020-08-08 10:45:37 -07:00
Chris Lu
bd8bfdae07 refactoring 2020-08-08 10:18:43 -07:00
Chris Lu
ab6e5c0dc4 adjust error message 2020-08-08 10:18:32 -07:00
Chris Lu
828a5ae429 check signature only when auth is enabled 2020-08-08 09:11:40 -07:00
Chris Lu
20e2ac1add filer: store md5 metadata for files uploaded by filer 
fix https://github.com/chrislusf/seaweedfs/issues/1412
 2020-08-06 10:04:17 -07:00
Chris Lu
93ea0801ea volume: the variable for the master node may be stale? 
related to https://github.com/chrislusf/seaweedfs/issues/1414
 2020-08-06 09:48:54 -07:00
Chris Lu
4ecfa9879d volume: report Content-MD5 in response header 2020-08-06 05:22:53 -07:00
Chris Lu
41007ced77 remove logging 2020-08-06 04:26:29 -07:00
Chris Lu
2b74abf766 S3: configurable access for anonymous user 
fix https://github.com/chrislusf/seaweedfs/issues/1413
 2020-08-06 03:41:34 -07:00
Устюжанин Антон Александрович
9ca011e3ca ListDirectoryPrefixedEntries 2020-08-06 00:38:59 +05:00
Устюжанин Антон Александрович
b231f7bdab ListDirectoryPrefixedEntries 2020-08-06 00:37:42 +05:00
Устюжанин Антон Александрович
a457c308ad test ListDirectoryPrefixedEntries 2020-08-06 00:24:31 +05:00
Устюжанин Антон Александрович
2d3b355fb6 test ListDirectoryPrefixedEntries 2020-08-05 23:56:06 +05:00
Устюжанин Антон Александрович
67f134ebd7 test ListDirectoryPrefixedEntries 2020-08-05 23:44:48 +05:00
Устюжанин Антон Александрович
dc9fc01254 test ListDirectoryPrefixedEntries 2020-08-05 23:38:00 +05:00
Устюжанин Антон Александрович
33a9e5e2d1 test ListDirectoryPrefixedEntries 2020-08-05 22:19:16 +05:00
Chris Lu
cbd80253e3 better needle id format 2020-08-04 22:09:07 -07:00
Chris Lu
1b3a80dd3d non-fatal error 2020-08-01 13:46:52 -07:00
Chris Lu
bbbea8159c http request use gzip if possible 2020-08-01 13:20:52 -07:00
Chris Lu
2f03481cb2 in case when content is not compressed 2020-08-01 13:08:10 -07:00
Chris Lu
49199c69d4 volume: avoid special logic for .gz files 2020-08-01 11:19:57 -07:00
Chris Lu
f89186a3cf better logging 2020-08-01 11:19:57 -07:00
Chris Lu
49d6dcfbc6 dynamically change the log level for reading the disk status 2020-08-01 11:19:57 -07:00
limd
b41b7ea4d0 fix spark read s3 bug (sc.binaryFiles) 2020-08-01 01:08:30 +08:00
Chris Lu
a6b59d50f7 1.87 2020-07-30 13:52:04 -07:00
Chris Lu
eed525b717 FUSE mount: remove DirListCacheLimit 
outdated parameter
 2020-07-28 09:24:39 -07:00
Chris Lu
a566bfc6e1 s3: use bucket in the domain 
fix https://github.com/chrislusf/seaweedfs/issues/1405
 2020-07-28 08:47:27 -07:00
limd
92c32f9d46 PutObject: fix create folder bug 2020-07-28 00:58:42 +08:00
Chris Lu
54cd582653 refactor 2020-07-26 12:58:58 -07:00
Chris Lu
c50ec1599c auth for presigned put object part 2020-07-26 12:09:50 -07:00
Chris Lu
ea0180de4e auth for presigned put operation 2020-07-26 12:06:45 -07:00
Chris Lu
3faa0aa448 rename 2020-07-25 20:09:19 -07:00
Chris Lu
023a1efdf2 set filename in Content-Disposition header 2020-07-25 20:06:40 -07:00
popstk
a09fcfcf20 fix s3api auth bug 2020-07-25 17:14:49 +08:00
popstk
feb52400be fix s3api copy object handler SerializationError 2020-07-25 15:52:31 +08:00
Chris Lu
d40de39e75 FUSE: do not change crtime, uid, gid on save 2020-07-24 10:13:45 -07:00
Chris Lu
8dfeba8023 FUSE: skip changing to empty uid and gid during flush 
Skip uid and gid changes if request uid and gid are zero

mongodb lock file, and many interim files, has empty uid and gid

I0724 10:05:51 93643 filehandle.go:173] /db/diagnostic.data/metrics.interim fh 1333342842031408359 flush Flush [ID=0x3 Node=0x50 Uid=0 Gid=0 Pid=178] 0x10 fl=0x0 lk=0x0
 2020-07-24 10:06:43 -07:00
Chris Lu
dd29b8c81d fix compilation 2020-07-24 09:36:05 -07:00
Chris Lu
5788f1f6bf remove fsnode cache 
fix https://github.com/chrislusf/seaweedfs/issues/1400
 2020-07-24 09:25:54 -07:00
Chris Lu
6962ca48d3 fix compilation 2020-07-23 21:31:23 -07:00
Chris Lu
df7eee523b FUSE: implement dir fsync 
fix https://github.com/chrislusf/seaweedfs/issues/1398
 2020-07-23 21:09:40 -07:00
Chris Lu
3bbac0820b FUSE: faster rename implementation 2020-07-23 21:08:42 -07:00
Chris Lu
943fa4ea0c fix compilation error 2020-07-22 16:11:17 -07:00
Chris Lu
49446af516 simplify 2020-07-22 13:43:54 -07:00
James Hartig
229f11c660 Added VolumeNeedleStatus volume server grpc method 
This is needed for the diffing tool to get the cookie for a needle
 2020-07-22 15:02:21 -04:00
James Hartig
0be3bb4a87 idx: Switch WalkIndexFile to accept generic io.ReaderAt 
This is necessary to use it with in-memory idx files.
 2020-07-22 15:01:16 -04:00
Chris Lu
7a7e70b6cc 1.86 2020-07-20 23:39:58 -07:00
Chris Lu
885c624bce volume.fsck: follow manifest chunks 2020-07-20 22:02:05 -07:00
Chris Lu
44057a4de1 clean up chunks in manifest 2020-07-20 22:01:39 -07:00
Chris Lu
0b2e06268b use merge factor 1000 2020-07-20 18:30:09 -07:00
Chris Lu
d02c0fe0c0 refactoring 2020-07-20 03:34:06 -07:00
Chris Lu
60d14a9800 mount: fix difference with storage format in local cache 2020-07-20 00:17:50 -07:00
Chris Lu
97d97f3528 go code can read and write chunk manifest 2020-07-19 17:59:43 -07:00
李远军
4162766c46 Fix the bug of "http://xx/xx,xxx.PNG?width=100" 
If you want to do a resize a UPPER ext pic file, there is a bug caused by commit 308688c8d0 (also see https://github.com/chrislusf/seaweedfs/blob/master/weed/images/resizing.go#L43)
 2020-07-17 19:57:35 +08:00
Chris Lu
f43146b237 resolve directories if containing home directory 2020-07-16 22:50:14 -07:00
Chris Lu
befb396892 mount: resolve home directory 
fix https://github.com/chrislusf/seaweedfs/issues/1391
 2020-07-16 22:38:55 -07:00
Chris Lu
2b2ca7d1dc stop if the directory can not be found 
related to https://github.com/chrislusf/seaweedfs/issues/1391
 2020-07-16 13:24:53 -07:00
Chris Lu
2286d27730 rename 2020-07-15 14:19:15 -07:00
Chris Lu
0db4204c81 add is_file_chunks to filer.proto 2020-07-14 22:42:52 -07:00
Chris Lu
bbc8ae110f report errors 2020-07-14 11:27:00 -07:00
Chris Lu
17f2d05cde filer: ensure logs are persisted 2020-07-14 11:25:50 -07:00
Chris Lu
ddec7b2bb9 go fmt 2020-07-14 07:34:16 -07:00
Chris Lu
4f6096c7f0 add reading from persisted logs for local filer store 2020-07-13 22:55:28 -07:00
Chris Lu
87b5031714 change variable visibility 2020-07-13 08:19:48 -07:00
Chris Lu
7597831cac filer: leveldb2 supports peers also 2020-07-13 00:05:40 -07:00
Chris Lu
5ba894bb72 leveldb2: support subscribing from peers 2020-07-13 00:05:40 -07:00
Chris Lu
5a4f258a8c 1.85 2020-07-12 17:34:09 -07:00
Chris Lu
4eecc6abc6 filer: emit metadata notification correctly for batch deletion 2020-07-12 17:32:19 -07:00
Chris Lu
1dd2c76532 refactoring 2020-07-12 17:31:24 -07:00
Chris Lu
224103e13b aggregated logs has empty flushFn 2020-07-11 09:12:03 -07:00
Chris Lu
bfabb4159b refactoring 2020-07-11 09:11:15 -07:00
Chris Lu
f2e8ad6241 refactoring 2020-07-11 06:35:54 -07:00
Chris Lu
2d1df97933 mount: fatal error if can not process meta data 2020-07-11 06:16:48 -07:00
Chris Lu
e3e00d36c5 reset cache for each new release 2020-07-11 06:16:17 -07:00
Chris Lu
7ad0bce2f5 remove old code 2020-07-10 23:03:22 -07:00
Chris Lu
e4ef3814e7 fix compilation 2020-07-10 10:02:59 -07:00
Chris Lu
e7c04af1d0 skip http.StatusPartialContent in case of error and superfluous response.WriteHeader 2020-07-10 08:56:23 -07:00
李远军
8c318470dd Revert "remove fixJpgOrientation" 
This reverts commit de5ca9b2

minimise the revert changes
 2020-07-10 10:28:28 +08:00
李远军
6608cb5f43 Revert "remove fixJpgOrientation" 
This reverts commit de5ca9b2
 2020-07-10 10:08:36 +08:00
cheng.li01
dad1161c70 fix dn.volumes Iterate when write issue 2020-07-08 19:57:19 +08:00
Chris Lu
54d6b3c30f filer: support running multiple mount to different filers avoiding duplicated cache 
fix https://github.com/chrislusf/seaweedfs/issues/1382
 2020-07-07 23:18:44 -07:00
Chris Lu
4bed1663e2 adjust log level 2020-07-07 23:17:31 -07:00
Chris Lu
482d13468b remove println 2020-07-07 23:17:17 -07:00
Chris Lu
a2eb680f34 filer: default filer store directory 
1. set default filer store directory
2. set peers, avoiding empty string counted as 1.
 2020-07-07 23:06:48 -07:00
Chris Lu
3c269da37f fix compilation 2020-07-06 09:46:24 -07:00
Chris Lu
d198e8c6d6 filer: support cross filer meta data sync if sharing the same store 2020-07-06 09:24:54 -07:00
Chris Lu
0b8cc9b626 aggregate multiple filer metadata chagne events 2020-07-05 23:05:02 -07:00
Chris Lu
49929e0869 rename 2020-07-05 15:52:36 -07:00
Chris Lu
70d8a3a1d3 add SubscribeLocalMetadata without checking persisted meta logs 2020-07-05 15:50:07 -07:00
Chris Lu
55e40b08fc refactoring 2020-07-05 15:43:06 -07:00
Chris Lu
881e0fde2e rename 2020-07-05 08:55:09 -07:00
Chris Lu
353bea8ddb refactoring 2020-07-03 22:25:35 -07:00
Chris Lu
ce3630e7a4 refactoring 2020-07-03 22:16:59 -07:00
Chris Lu
59152334c7 default to 1% 2020-07-03 16:42:53 -07:00
Chris Lu
acf4a9ea99 volume: min free space default to 1% 2020-07-03 16:42:02 -07:00
Chris Lu
4d1484628a refactoring 2020-07-03 16:41:30 -07:00
Chris Lu
faa5c2e89a refactoring 2020-07-03 16:34:31 -07:00
Chris Lu
37d5b3ba12 replication: pass isFromOtherCluster also to EventNotification 
EventNotification is consistent with message queue and metadata logs.
 2020-07-01 08:06:20 -07:00
Chris Lu
31e23e9783 filer: support active<=>active filer replication 2020-06-30 22:53:57 -07:00
Chris Lu
df75b5c98d passing ctx for metadata logging 2020-06-28 14:34:51 -07:00
Chris Lu
12f6b683fc default to 8 volumes 
in case all the volumes are allocated for a collection
 2020-06-28 14:34:21 -07:00
Chris Lu
42a338d7b0 FUSE mount: clean up file handles during renaming 
resolve dir rename when file is still open. Need to clean the file handles as soon as possible.

These can happen out of order:
file rename, then file release
file release, then file rename
 2020-06-28 13:41:00 -07:00
Chris Lu
95a1860d65 remove empty calls for list cache 2020-06-28 10:25:54 -07:00
Chris Lu
6498d0c86b remove list cache 2020-06-28 10:21:52 -07:00
Chris Lu
9033a7d369 removed async option 2020-06-28 10:18:32 -07:00
Chris Lu
1bb8cae65d reverting and working 2020-06-28 10:14:17 -07:00
Chris Lu
b813fac4a3 1.84 2020-06-28 07:28:21 -07:00
Chris Lu
20d9cfae86 volume: following md5 base64 encoding 
fix https://github.com/chrislusf/seaweedfs/issues/1374
 2020-06-28 07:25:50 -07:00
Chris Lu
f0f884edd5 1.83 2020-06-27 15:46:59 -07:00
Chris Lu
c2797cc6de add logs 2020-06-27 15:46:53 -07:00
Chris Lu
62260e526b add logging 2020-06-27 12:54:37 -07:00
Chris Lu
bef356ce4c since we already know the chunk size, no need to iterate 2020-06-27 12:51:04 -07:00
Chris Lu
a808b3b5df incase the memory data is too small 2020-06-27 11:59:15 -07:00
Chris Lu
3dbd51c3c2 a little bit more efficient 2020-06-26 10:02:37 -07:00
Chris Lu
212b6e7d42 error logging 2020-06-26 10:01:55 -07:00
Chris Lu
48b23f2fdd FUSE mount: prevent concurrent modification 2020-06-26 10:00:48 -07:00
Chris Lu
3cec4b3c49 FUSE mount: fix file id written twice? 
fix https://github.com/chrislusf/seaweedfs/issues/1373
 2020-06-25 20:11:49 -07:00
Chris Lu
212c64241b checking ecx file 2020-06-25 16:36:06 -07:00
Chris Lu
a4dc25cd22 volume: generate ec shard first, before generating the ec index file. 
possible fix https://github.com/chrislusf/seaweedfs/issues/1364

the ec index file is used to determine whether the ec files are ready or not
 2020-06-25 11:06:58 -07:00
Chris Lu
9fd7cdadf1 fix 2020-06-25 10:45:34 -07:00
Chris Lu
3b638d3994 add more ec encoding logging 2020-06-25 09:43:38 -07:00
Chris Lu
e8e0d629af ensure correct file mode 2020-06-24 23:26:54 -07:00
Chris Lu
32a1f22730 more logs 2020-06-24 22:07:53 -07:00
Chris Lu
5be12eea37 zstd fix 2020-06-24 11:39:12 -07:00
Chris Lu
2ff37ccdbd testing parsing the upload 
The mime type is always the value passed in.

Compress or not depends on the content detection, file name extension, and compression ratio.

If the content is already compressed, need to know the content size.
 2020-06-24 11:39:12 -07:00
Chris Lu
c21f4ebfee
Merge pull request #1371 from Kimbsen/content_md5_validation 
Optional md5 validation of uploads
 2020-06-24 11:33:44 -07:00
Kimbsen
4eeab2a379 remove unused imports 2020-06-24 13:37:00 +02:00
Kimbsen
ffddecebef do md5 validation AFTER decompression 2020-06-24 13:35:13 +02:00
Chris Lu
fe60db404a prepare for zstd 2020-06-23 09:12:02 -07:00
Kimbsen
6b1e93ba0b check request header not part header for content-encoding 2020-06-23 17:04:24 +02:00
Kimbsen
38626cb584 don't forget the last few bytes 2020-06-23 16:53:49 +02:00
Kimbsen
8d1f6cd567 added optional md5 verification of uploaded data 2020-06-23 16:24:11 +02:00
Chris Lu
a4e3cffe0a S3: DeleteObject API also support deleting a folder recursively 2020-06-22 10:01:00 -07:00
Chris Lu
98493e0152 volume: add default minFreeSpacePercent 
fix https://github.com/chrislusf/seaweedfs/issues/1366
 2020-06-21 08:44:06 -07:00
Chris Lu
e7c36f740a refactoring 2020-06-21 08:38:00 -07:00
Chris Lu
8b4a32e782 adjust help message 2020-06-21 08:34:24 -07:00
Chris Lu
b4ec04d464 fix the help message 2020-06-21 08:31:20 -07:00
Chris Lu
de5ca9b258 remove fixJpgOrientation 2020-06-20 12:50:40 -07:00
Chris Lu
3080c197e3 rename UnCompressData to DecompressData 2020-06-20 08:16:16 -07:00
Chris Lu
97239ce6f1 rename filechunk is_gzipped to is_compressed 2020-06-20 08:15:49 -07:00
Chris Lu
a735fd767c adjust uncompress data 2020-06-20 08:01:00 -07:00
Chris Lu
ca3516ac6d adjust protoc 2020-06-20 08:00:25 -07:00
Chris Lu
e912fd15e3 renaming 2020-06-19 22:45:27 -07:00