mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-12 08:42:01 +08:00
Code Issues Packages Projects Releases Wiki Activity
4,788 Commits 32 Branches 296 Tags 150 MiB
d9e8479c06
Commit Graph

3007 Commits

Author SHA1 Message Date
Chris Lu
3bf0116de1 mount: less channel waiting 2020-10-21 02:16:21 -07:00
Chris Lu
c31b254248 mount: shortcut when there is only one chunk 2020-10-20 23:48:29 -07:00
Chris Lu
f64252023e Revert "a better byte buffer pool" 
This reverts commit 59c6422777.
 2020-10-20 23:21:54 -07:00
Chris Lu
ad652ffb97 avoid close closed channel 2020-10-20 22:54:34 -07:00
Chris Lu
20442dc1bc avoid extra data copying 2020-10-20 22:54:21 -07:00
Chris Lu
59c6422777 a better byte buffer pool 2020-10-20 22:53:58 -07:00
Chris Lu
6c92fa349d filer: support default rack 
fix https://github.com/chrislusf/seaweedfs/issues/1546
 2020-10-20 17:41:39 -07:00
Chris Lu
b3aa2fab9a s3: report error to s3 when updating an object but it is already a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1545
 2020-10-20 10:25:16 -07:00
Chris Lu
410b818aa7 master: avoid timer leakage 2020-10-19 14:24:57 -07:00
Chris Lu
69f0da5d73 mount: pass along replication and collection parameters 2020-10-19 00:12:42 -07:00
Chris Lu
c0ab458671 report nil instead of EOF for empty files 
related to https://github.com/chrislusf/seaweedfs/issues/1541
 2020-10-17 11:03:46 -07:00
Chris Lu
898db14729 2.05 2020-10-17 02:42:33 -07:00
Chris Lu
09bab17aff mount: avoid "send on closed channel" 2020-10-16 12:54:23 -07:00
Chris Lu
9d80a3428c add debug_webdav 2020-10-16 12:53:56 -07:00
Chris Lu
37f165d743 webdav: return io.EOF when at end of a file 
fix https://github.com/chrislusf/seaweedfs/issues/1344
 2020-10-16 12:53:37 -07:00
Chris Lu
68d39c86f1 mysql, postgres, cassandra: change kv key to base64 encoding 
The exisitng key-value operation for stores using mysql, postgres, and maybe cassandra are already broken.

The kv is used to store hardlink, filer store signature and replication progress.

So users using hardlink and also uses mysql, postgres, or cassandra will have broken hard links.

Users using filer.sync will need to re-sync the files.
 2020-10-16 11:10:12 -07:00
Chris Lu
ee1fc6558a refactor 2020-10-16 11:02:44 -07:00
Chris Lu
06e6341097 set initial write time to avoid race condition with delayed chunks 2020-10-15 11:08:45 -07:00
Chris Lu
d598c47136 log errors 2020-10-15 11:01:43 -07:00
Chris Lu
5039aff310 release the goroutine to read errors 2020-10-15 10:59:18 -07:00
Chris Lu
ace0ea3d28 s3: avoid duplicated bucket 2020-10-15 10:52:20 -07:00
Konstantin Lebedev
dc2e13092d add number of read only volumes metric 2020-10-15 15:48:40 +05:00
Chris Lu
e714c28a02 mount: async write file chunk 2020-10-14 23:28:03 -07:00
Chris Lu
c95d3d9dff adjust tests 2020-10-14 12:27:52 -07:00
Chris Lu
93bcf56514 file read report EOF 
fix https://github.com/chrislusf/seaweedfs/issues/1344
 2020-10-14 12:18:24 -07:00
Chris Lu
1069b325dd shell: volumeServer.evacuate adds printout for ec volumes 2020-10-13 20:26:03 -07:00
Chris Lu
0542911e29 go fmt 2020-10-13 19:50:46 -07:00
Chris Lu
58fa506491 minor 2020-10-13 19:50:22 -07:00
Chris Lu
28d4e1a51b mount: retry for directory listing with filer 
related to https://github.com/chrislusf/seaweedfs/issues/1530
 2020-10-13 19:49:52 -07:00
Chris Lu
c127da1219 filer: linearize timeout for large chunk of data 2020-10-13 14:04:46 -07:00
Chris Lu
aac4cb1f0c adds errror on read and write 2020-10-13 13:53:34 -07:00
Chris Lu
9b4f7fed14 mount: report filer IO error 
related to https://github.com/chrislusf/seaweedfs/issues/1530
 2020-10-13 11:21:13 -07:00
Chris Lu
3f7d1d1bf1 Only wait on retryable requests 2020-10-13 00:29:46 -07:00
Chris Lu
b18f21cce1 mount: fix bound tree with filer.path 
fix https://github.com/chrislusf/seaweedfs/issues/1528
 2020-10-12 21:58:37 -07:00
Chris Lu
f022aff289 add back http.StatusPartialContent 
revert e7c04af1d0
 2020-10-12 12:26:25 -07:00
Chris Lu
bbd0afd37e 2.04 2020-10-11 21:25:30 -07:00
Chris Lu
7704469d29 mount: outsideContainerClusterMode changed to use volume server publicUrl 2020-10-11 20:42:15 -07:00
Chris Lu
723ae11db4 refactoring in order to adjust volume server url later 2020-10-11 20:15:10 -07:00
Chris Lu
d155f907c2 mount: configurable read wait time 2020-10-10 20:09:43 -07:00
Chris Lu
9b0e8ef026 filer: added QR code to transfer files with mobile devices 2020-10-10 18:00:21 -07:00
Chris Lu
8a52379ecb add retry if volume can not be found 2020-10-10 16:02:39 -07:00
Chris Lu
70af0ec24c filer: fix hanging on read 2020-10-10 16:02:10 -07:00
Chris Lu
cff8bb6554 return proper error 2020-10-10 15:43:22 -07:00
Chris Lu
6da8eef54f fix logic error 2020-10-09 00:01:47 -07:00
Chris Lu
5d01dd28a6 add logging 2020-10-08 23:38:00 -07:00
Chris Lu
8d34eb0050 mount:exponentially backoff if read error for about 10 minutes 2020-10-08 23:31:26 -07:00
Chris Lu
6e1f936efd refactoring 2020-10-08 23:19:42 -07:00
Chris Lu
b2ee5873fb fix error not being returned 2020-10-08 23:19:20 -07:00
Chris Lu
b067853162 mount: ignore recursion error 
fix https://github.com/chrislusf/seaweedfs/issues/1514

the host OS should manage recursive deletion.
 2020-10-08 21:55:55 -07:00
Chris Lu
e1a8e3f900 fix logging 2020-10-08 18:40:38 -07:00
Chris Lu
8676db0f6a adjust logging 2020-10-08 18:33:37 -07:00
Chris Lu
bd8b9b0c2e some changes to prepare fix after failed reads 2020-10-08 18:33:06 -07:00
Chris Lu
78859e35fc s3: adjust config examples 2020-10-08 10:12:09 -07:00
Chris Lu
9e7a2772b1 s3: only admin can list all buckets 2020-10-08 10:12:09 -07:00
Chris Lu
eed492b73b randomize file locations 2020-10-07 23:58:32 -07:00
Chris Lu
ec08a1670b volume: avoid hanging when stopping a volume server 2020-10-07 23:48:24 -07:00
Chris Lu
62d3d3aea0 go fmt 2020-10-07 23:30:54 -07:00
Chris Lu
e91b9c85a2 s3: Added support for "List" action in weed s3 -config=... in the config file. 
fix https://github.com/chrislusf/seaweedfs/issues/1511
 2020-10-07 23:22:35 -07:00
Chris Lu
a8624c2e4f read from alternative replica 
related to https://github.com/chrislusf/seaweedfs/issues/1512
 2020-10-07 22:49:04 -07:00
Chris Lu
da4edf3651 master: check peers for existing leader before starting a leader election 
fix https://github.com/chrislusf/seaweedfs/issues/1509
 2020-10-07 01:25:39 -07:00
Chris Lu
c543762e23 volume: auto expands -max setting 
I have tried to run weed volume but got errors...
cmd:
weed volume -max=0 -mserver="127.0.0.1:9333" -port=8080 -dir="/var/www/d1/sfs,/var/www/d2/sfs,/var/www/d3/sfs"
log:
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d1/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d2/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d3/sfs Permission: -rwxr-xr-x
F1006 15:09:26 17040 volume.go:142] 3 directories by -dir, but only 1 max is set by -max
What am I doing wrong? Thx.
 2020-10-06 09:05:30 -07:00
Chris Lu
36492c47ec adjust 2020-10-05 14:06:18 -07:00
Chris Lu
8f8e9ddbde adjust API 2020-10-05 09:47:07 -07:00
Konstantin Lebedev
e4f2d9eb4a We return etag using the same algorithm as aws s3 
https://teppen.io/2018/06/23/aws_s3_etags/
 2020-10-05 14:43:32 +05:00
Chris Lu
4fc673341f 2.03 2020-10-04 21:58:22 -07:00
Chris Lu
6b591b02af adjust log level 2020-10-04 16:21:43 -07:00
Устюжанин Антон Александрович
702b1cb876 fix: remove deleted peers if resumeState = true 2020-10-04 21:56:17 +05:00
Chris Lu
d522df0ecd adjust log level 2020-10-04 01:35:32 -07:00
Chris Lu
666859f7cd mount: read prefetching 2020-10-04 01:31:04 -07:00
Chris Lu
bc47835997 mount: avoid duplicated reads 2020-10-03 20:16:42 -07:00
Chris Lu
8d65ad1444 Revert "mount: adds read prefetching" 
This reverts commit 692f0614d1.
 2020-10-03 19:40:56 -07:00
Chris Lu
692f0614d1 mount: adds read prefetching 2020-10-03 16:50:09 -07:00
Chris Lu
707936f482 re-enable caching larger than 16MB 
revert 62ce85610e
 2020-10-03 14:12:38 -07:00
Chris Lu
2c20ef72ae fix starting problem 2020-10-03 13:56:38 -07:00
Chris Lu
d77e3c96e9 FUSE: use the mount directory permissions 2020-10-03 13:37:35 -07:00
Chris Lu
95ffed87c0
Merge pull request #1505 from ustuzhanin/remove_raft_state 
Resume raft state
 2020-10-03 10:38:47 -07:00
Устюжанин Антон Александрович
6fbfe73683 fix: resumeState 2020-10-03 22:16:47 +05:00
Устюжанин Антон Александрович
dc31b19469 fix: restore raft state 2020-10-03 14:03:41 +05:00
Chris Lu
f781cce500 s3: support object tagging 
* GetObjectTagging
* PutObjectTagging
* DeleteObjectTagging
 2020-10-02 22:21:51 -07:00
Устюжанин Антон Александрович
8c82fb7e5f fix: restore raft state 2020-10-02 23:01:20 +05:00
Chris Lu
9ab98fa912 s3 metrics adjust the label 2020-10-01 07:17:47 -07:00
Chris Lu
f1e879fe7a 2.02 2020-10-01 07:15:49 -07:00
Chris Lu
a1c01d716b volume: avoid deadlock when deleting volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1501
 2020-10-01 07:10:03 -07:00
Konstantin Lebedev
86329bbf2b label name is statusCode 2020-10-01 01:22:38 +05:00
Konstantin Lebedev
68463e92c1 add status code in S3RequestCounter 2020-10-01 00:59:39 +05:00
Chris Lu
79ab10e300 adjust help message 2020-09-30 09:32:00 -07:00
Chris Lu
55cb68c09c filer: replication follows master config if not specified 2020-09-30 09:15:55 -07:00
Chris Lu
b9887504e8 fix test 2020-09-27 23:19:50 -07:00
Chris Lu
f46eae284e adjust for test 2020-09-27 23:08:11 -07:00
Chris Lu
75aca5e13d 2.01 2020-09-27 23:02:41 -07:00
Chris Lu
e6552b5e1e filer: able to start s3 together 2020-09-27 23:00:43 -07:00
Chris Lu
474e2b6ac3 add a hard link marker to 16byte + maker, for future extensions 2020-09-27 22:38:30 -07:00
Chris Lu
c49e2bb9a3 adjust 2020-09-27 12:07:45 -07:00
Chris Lu
62ce85610e skip caching too large chunks 2020-09-27 11:58:48 -07:00
Chris Lu
9ad2dcca2b more tests 2020-09-27 11:42:51 -07:00
Chris Lu
e43d86c796 fix pre allocated volume size 2020-09-27 10:58:19 -07:00
Chris Lu
31fc7bb2e1 refactor 
adjust for faster test
 2020-09-27 10:41:29 -07:00
Chris Lu
a37535cd9f avoid non utf-8 in filename 
fix https://github.com/chrislusf/seaweedfs/issues/1493
 2020-09-26 00:13:39 -07:00
Chris Lu
41be7a4c8f filer: upload also set file size 
ensure works same as fuse mount
 2020-09-25 12:02:06 -07:00
Chris Lu
35f6518c36 weed upload: usePublicUrl was not being used on big files 
fix https://github.com/chrislusf/seaweedfs/issues/1492
 2020-09-25 02:41:38 -07:00
Chris Lu
22049dc2e6
Merge pull request #1491 from hilimd/master 
s3:  fix get object metadata problem
 2020-09-25 02:35:00 -07:00
limd
402aef8f30 s3: 
1.fix spark reading S3 directory wildcard problem
2.fix the problem of the spark history service writing S3 directory

reference git revsion number: b41b7ea4d0
 2020-09-25 14:37:02 +08:00
Chris Lu
8516517c48 filer store: Cassandra supports username/password 2020-09-24 21:31:06 -07:00
limd
ed7816681a Add cassandra authenticator mode 2020-09-25 11:22:47 +08:00
limd
370a98cf6b code style 2020-09-25 11:19:42 +08:00
limd
aee3fd08b9 code style 2020-09-25 11:04:32 +08:00
Chris Lu
043b063136
Merge pull request #1482 from hilimd/master 
Fix: s3 delete object
 2020-09-24 18:21:34 -07:00
limd
48c578410f mount: rollback default value 2020-09-25 09:18:52 +08:00
Chris Lu
dbf5327b98 s3: handle response-content-encoding 
fix https://github.com/chrislusf/seaweedfs/issues/1487
 2020-09-24 18:09:52 -07:00
Chris Lu
7726965a47 minor 2020-09-24 11:20:12 -07:00
Chris Lu
0790c6d605 fix empty hard link id 2020-09-24 11:16:43 -07:00
Chris Lu
1012df7bb5 switch hardlink id from int64 to bytes 2020-09-24 11:11:42 -07:00
Chris Lu
4856bce0ee adjust for metrics port 2020-09-24 10:21:23 -07:00
Chris Lu
d2d3aec3e1 consolidate to one metricsPort in "weed server" mode 2020-09-24 09:55:02 -07:00
Chris Lu
2e9099369e
Merge pull request #1489 from kmlebedev/promhttp 
Promhttp
 2020-09-24 09:50:48 -07:00
Chris Lu
2e7c361a0d hardlink deletion factors in hardlink counter 2020-09-24 09:43:52 -07:00
Chris Lu
1295347958 adjust hardlink update 
simplify logic, pass entity content directly to hard link. The "weed mount" handles the logic to calculate hard link counter.
 2020-09-24 09:43:00 -07:00
Konstantin Lebedev
98e9de6e11 fix style 2020-09-24 17:48:39 +05:00
Konstantin Lebedev
324e44d4b3 add start metrics server 2020-09-24 17:45:39 +05:00
limd
cbca14edc5 mount: fix k8s pvc and os mount directory permission bug 2020-09-24 18:07:16 +08:00
Chris Lu
5e239afdfc hardlink works now 2020-09-24 03:06:48 -07:00
limd
59e91e9c7e mount: fix k8s pvc mount directory permission 2020-09-24 10:25:45 +08:00
Chris Lu
c7d7b1a0f6
Merge pull request #1485 from LIBA-S/fix_oversized 
Correct the oversized state of volume after compaction
 2020-09-23 19:24:30 -07:00
Chris Lu
b80a2b3cc9
Merge pull request #1486 from LIBA-S/fix_race_condition 
Fix a race condition when handle VolumeLocationList
 2020-09-23 19:24:16 -07:00
LIBA-S
eecd6b5d35 Fix a race condition when handle VolumeLocationList 2020-09-23 20:56:51 +08:00
LIBA-S
0157798ebf Correct the oversized state of volume after compaction 2020-09-23 20:27:42 +08:00
limd
9b21ec27e4 mount: auto created dir set corrct umask fix bug 2020-09-23 18:35:37 +08:00
hilimd
0db149fb5f
Merge pull request #18 from chrislusf/master 
sync
 2020-09-23 17:32:05 +08:00
Chris Lu
f7a0ccb595 mount: auto created dir set corrct umask 2020-09-23 02:31:19 -07:00
Chris Lu
f0e325b6fc mount: auto created directory follow umask 2020-09-23 02:27:57 -07:00
limd
d506080c36 rollback 2020-09-23 14:29:53 +08:00
limd
8f9f29b773 fixed the problem of empty directory when S3 deleted the directory 2020-09-23 13:33:13 +08:00
limd
1892677b22 fixed the problem of empty directory when S3 deleted the directory 2020-09-23 13:15:06 +08:00
hilimd
85570f27a0
Merge pull request #17 from chrislusf/master 
sync
 2020-09-23 11:39:57 +08:00
limd
a99f63cb4d 1.add S3 copy directory function 
2.fixed the problem of empty directory when S3 deleted the directory
 2020-09-23 11:26:01 +08:00
Chris Lu
b61d33f251 expose only store wrapper to meta changes 2020-09-22 16:27:36 -07:00
Chris Lu
0adbb56cc1 rename 2020-09-22 16:24:13 -07:00
Chris Lu
3e52329cee Revert "Merge pull request #1479 from LIBA-S/fix_oversized" 
This reverts commit bd11f0b3e4, reversing
changes made to ec5b9f1e91.
 2020-09-22 15:05:37 -07:00
Chris Lu
4a1fe4b8e2 add error logs 2020-09-22 09:16:10 -07:00
LIBA-S
8573ac82c6 Fix: remove the oversized state after compaction 2020-09-22 21:48:39 +08:00
James Hartig
658fc2e5b6 Allow option to enable volume pprof on server 2020-09-21 22:43:10 -04:00
James Hartig
91e4eca1e9 Fix deadlock with KeepConnected and SendHeartbeat 
There's the potential where we're writing to a clientConn and it goes away
and we're stuck keeping a read lock on clientChansLock. This causes
KeepConnected to not be able to remove the client since it requires a write
lock on clientChansLock. This ends up backing up SendHeartbeat because it
can't get a read lock.
 2020-09-21 22:41:38 -04:00
Chris Lu
9cdbfc1a49 refactor 2020-09-21 17:34:38 -07:00
Chris Lu
63373a9f9f filer: add file extended properties 2020-09-21 11:08:34 -07:00
Chris Lu
b9c1f3e9de s3: fixes for list multipart upload 2020-09-21 10:51:24 -07:00
Chris Lu
289e62a305 master: better locking of in memory volume data 
related to https://github.com/chrislusf/seaweedfs/issues/1436#issuecomment-695880135
 2020-09-20 23:07:55 -07:00
Chris Lu
9a3b564508 adjust printout 2020-09-20 16:01:56 -07:00
Chris Lu
62563a895a refactoring 2020-09-20 16:00:01 -07:00
Chris Lu
c2faab23b6 refactor 2020-09-20 15:40:49 -07:00
Chris Lu
d013d09a9b adjust logging 2020-09-20 15:38:59 -07:00
Chris Lu
f498c71199 shell: move volume operations to use flag parsing arguments 2020-09-20 09:27:34 -07:00
Chris Lu
fcbc520373 fix print out 2020-09-20 08:47:30 -07:00
Chris Lu
56094541c8 shell: adjust command option for collection.delete 2020-09-20 08:46:16 -07:00
Chris Lu
9cac9c4001 2.00 2020-09-20 00:47:04 -07:00
Chris Lu
dc4d2145ff open up 2020-09-19 20:23:12 -07:00
Chris Lu
29abe980df s3: add support for PostPolicy 
fix https://github.com/chrislusf/seaweedfs/issues/1426
 2020-09-19 20:14:19 -07:00
Chris Lu
41d508edfd go fmt 2020-09-19 14:10:26 -07:00
Chris Lu
5b40a2690a refactoring 2020-09-19 14:09:58 -07:00
Chris Lu
2c21eb1971 volume: get metrics configuration from master 
fix https://github.com/chrislusf/seaweedfs/issues/1354
 2020-09-19 00:03:00 -07:00
Chris Lu
2cbd1cf121 fix compilation 2020-09-18 00:15:54 -07:00
Chris Lu
23e9ede068 s3: collect metrics 2020-09-18 00:09:04 -07:00
Chris Lu
852e5f7cbc filer: fix mongodb insert 
fix https://github.com/chrislusf/seaweedfs/issues/1471
 2020-09-17 21:50:52 -07:00
Chris Lu
6544e60bea s3 add metrics 
empty for now
 2020-09-17 06:56:15 -07:00
Chris Lu
cb427d48fa filer report metrics configuration 2020-09-17 06:46:51 -07:00
Chris Lu
e4e0234998 refactoring 2020-09-17 06:43:54 -07:00
Chris Lu
feca07bf96 Filer: ip bind 
fix https://github.com/chrislusf/seaweedfs/issues/1470
 2020-09-16 10:26:08 -07:00
Chris Lu
e861a6a3ab simplify metrics configuration logic 2020-09-16 01:39:30 -07:00
Chris Lu
be54eeb364 passing value by checkWithMaster() instead 2020-09-16 01:33:45 -07:00
Chris Lu
4d21de63ee go fmt 2020-09-16 01:27:24 -07:00
Chris Lu
913a16268d volume: load configuration from master at the start 
fix https://github.com/chrislusf/seaweedfs/issues/1469
 2020-09-16 01:27:05 -07:00
Chris Lu
c9202c4b3d add storage backend to GetMasterConfigurationResponse 2020-09-16 01:12:22 -07:00
Chris Lu
8a0710cb73 handle more than 2GB files 
related to https://github.com/chrislusf/seaweedfs/issues/1468
 2020-09-16 00:41:50 -07:00
Chris Lu
854007bc98 minor 2020-09-16 00:37:57 -07:00
Chris Lu
10f9081526 filer: adjust meta data events to received timestamp 
if a client is already connected and start from t0. A message recieved at t+1 but with timestamp t-1 may not be processed by the client.

This commit changes to the event received time, so the replication can be ordered.
 2020-09-15 01:18:33 -07:00
Chris Lu
684a875876 update help message 2020-09-15 00:40:38 -07:00
Chris Lu
2d3a904a82 shell: volumeServer.evacuate adds option to skip non moveable volumes 2020-09-15 00:33:49 -07:00
Chris Lu
a595916342 shell: add volumeServer.evacuate command 2020-09-14 23:47:11 -07:00
Chris Lu
103fafe00b weed export: print out [start,stop) content range 2020-09-14 22:57:23 -07:00
Chris Lu
ada996fe55 filer: support createing empty folder 
fix https://github.com/chrislusf/seaweedfs/issues/1161
 2020-09-14 13:07:46 -07:00
Chris Lu
200fe5c83e go fmt 2020-09-13 21:26:30 -07:00
Chris Lu
5d6753fb98 shell: add volumeServer.leave command 2020-09-13 21:25:51 -07:00
Chris Lu
1af95c5b76 refactoring 2020-09-13 12:41:26 -07:00
Chris Lu
f8fea19669 1.99 2020-09-12 13:46:33 -07:00
Chris Lu
b0c7de186d filer: fix postgres prefixed directory listing problem 
fix https://github.com/chrislusf/seaweedfs/issues/1465
 2020-09-12 13:37:03 -07:00
Chris Lu
1a7afe7e6a adjust default value 
related to https://github.com/chrislusf/seaweedfs/issues/1453
 2020-09-12 12:46:42 -07:00
Chris Lu
f2723c1bc8 do not idx file format 
revert c9ab8d05fa
 2020-09-12 12:42:36 -07:00
Chris Lu
ba984a4e29 1.98 2020-09-12 04:13:02 -07:00
Chris Lu
446e476a11 go fmt 2020-09-12 04:08:03 -07:00
Chris Lu
ea26a98753 volume: validate volume correctness if last entry is a deletion 2020-09-12 04:07:04 -07:00
Chris Lu
d15682b4a1 shell: volume.balance plan by ratio of fullness 2020-09-12 04:06:26 -07:00
Chris Lu
c0ee78d2fa adjust make file 2020-09-12 04:05:42 -07:00
Chris Lu
cd9b89ba55 reduce default wait time to 10s before shutting down 2020-09-12 04:05:33 -07:00
Chris Lu
e2c741f76f adjust replica placement after move 2020-09-12 01:01:19 -07:00
Chris Lu
2a0925590c filer: etcd store avoid read with nil option 
fix https://github.com/chrislusf/seaweedfs/issues/1463
 2020-09-11 15:29:45 -07:00
Chris Lu
3984c3962f add comment 2020-09-11 15:07:19 -07:00
Chris Lu
baa6bdf4d4 s3: listMultipartUploads fix output format 2020-09-11 15:04:01 -07:00
Chris Lu
3eda8d6dfc s3: ListParts output xml format 
fix https://github.com/chrislusf/seaweedfs/issues/1461
 2020-09-11 14:53:50 -07:00
Chris Lu
ab201c2798 1.97 2020-09-11 13:47:50 -07:00
Chris Lu
9d4bdfcfdf fix volume integrity checking 2020-09-11 11:34:10 -07:00
Chris Lu
1b8094ef75 weed export: export deleted files 2020-09-11 02:05:14 -07:00
Chris Lu
12a8f5294d test for multi dc replication 2020-09-11 01:21:17 -07:00
Chris Lu
e60b2117c3 shell: volume balance follows replica placement 2020-09-11 00:29:25 -07:00
Chris Lu
89a62e8007 refactoring 2020-09-10 23:05:00 -07:00
Chris Lu
19537c9d21 1.96 2020-09-10 19:52:07 -07:00
Chris Lu
eaf9fdde99 avoid sharing context over separate goroutine 2020-09-10 19:51:03 -07:00
Chris Lu
cc5fe6f5ee handle special characters in html link 2020-09-10 19:46:02 -07:00
Chris Lu
401ccf1509 fix test 2020-09-10 15:13:23 -07:00
Chris Lu
c9ab8d05fa fixes for reading deleted fid 2020-09-10 14:42:52 -07:00
ruitao.liu
5b0676049a change elastic initialize process similar as others. 2020-09-10 23:35:20 +08:00
Chris Lu
660d7c0edd 1.95 2020-09-10 01:32:05 -07:00
Chris Lu
0d9b858cfb
Merge pull request #1455 from taozix/master 
add more basic elastic options.
 2020-09-10 01:31:17 -07:00
ruitao.liu
9be4e97625 change logs print format. 2020-09-10 16:30:15 +08:00
ruitao.liu
719dc43af1 modify elastic urls from string to array. 2020-09-10 16:24:09 +08:00
ruitao.liu
72f9d7f047 use util to generate md5. 2020-09-10 16:11:18 +08:00
ruitao.liu
6a5b38c0d4 fix elastic kv ops. 2020-09-10 15:59:16 +08:00
ruitao.liu
3f7fbfddca add more basic elastic options. 2020-09-10 14:22:07 +08:00
Chris Lu
b183ae54b5 master: changing 301 to 308 on redirect to volume servers 
fix https://github.com/chrislusf/seaweedfs/issues/1454
 2020-09-09 22:36:10 -07:00
Chris Lu
47b3f932e4 watch: adjust output format 2020-09-09 22:34:48 -07:00
Chris Lu
daf0a449f7 properly cancel context for streaming grpc 2020-09-09 12:07:15 -07:00
Chris Lu
7f69acd1f2 sync pprof 2020-09-09 11:33:52 -07:00
Chris Lu
387ab6796f filer: cross cluster synchronization 2020-09-09 11:21:23 -07:00
Chris Lu
4fc0bd1a81 return http response directly 2020-09-09 03:53:09 -07:00
Chris Lu
5e13bc878c adjust log level 2020-09-09 03:49:26 -07:00
Chris Lu
d1b816212f return nil if no need to do anything 2020-09-07 16:03:05 -07:00
Chris Lu
64a621bcc8 shell: volume.fix.replication also purge over replicated volumes 2020-09-07 16:00:10 -07:00
Chris Lu
c18ea21f7a 1.94 2020-09-07 13:13:53 -07:00
Chris Lu
18b98fdb72 fix test 2020-09-07 12:57:38 -07:00
Chris Lu
44b3f2efc8 filer: Elastic Search return correct kv error 
fix https://github.com/chrislusf/seaweedfs/issues/1452
 2020-09-07 12:44:51 -07:00
Chris Lu
d80538a187 refactoring 2020-09-07 12:35:02 -07:00
Chris Lu
1a09bc43d1 refactor 2020-09-07 11:31:33 -07:00
Chris Lu
2b643f477d typo 2020-09-06 12:47:55 -07:00
Chris Lu
05034aade5 printout over replicated locations 2020-09-06 12:44:02 -07:00
Chris Lu
432fe3d926 1.93 2020-09-06 12:12:09 -07:00
Chris Lu
d8af6f62df adjust logging 2020-09-06 12:10:40 -07:00
Chris Lu
07f32feed0 print sync progress 2020-09-06 00:50:51 -07:00
Chris Lu
0af6252b2d fix compilation 2020-09-06 00:50:38 -07:00
Chris Lu
ac494ff5e8 Update meta_aggregator.go 2020-09-06 00:29:16 -07:00
Chris Lu
c9f8f25ba5 read peer filer from start 2020-09-06 00:12:41 -07:00
Chris Lu
30dc365cbd add self filer 2020-09-06 00:11:46 -07:00
Chris Lu
8acd714600 apply meta changes only if store is different 2020-09-05 23:25:03 -07:00
Chris Lu
d741ed66db check whether shares the same filer store 2020-09-05 23:02:52 -07:00
Chris Lu
bba90ff3c8 read filer signature 2020-09-05 22:52:15 -07:00
Chris Lu
71b0e256c7 filer: setOrLoadFilerStoreSignature 2020-09-05 14:08:59 -07:00
ruitao.liu
450cf07505 skip the index that for kv usage. 2020-09-04 21:49:03 +08:00
ruitao.liu
1384ff9a2f 1.split kv in one file. 
2.disable query for kv in es index.
 2020-09-04 17:34:26 +08:00
ruitao.liu
83080b5e03 ES backended filer support kv ops. 2020-09-04 15:40:13 +08:00
Chris Lu
46f65a84a6 filer: elastic7 adjust default value 2020-09-03 20:12:38 -07:00
Chris Lu
f815fce81b
Merge pull request #1446 from taozix/master 
new filer store option to es7.
 2020-09-03 20:10:23 -07:00
Chris Lu
44b4ebf0ff filer: Redis cleanly delete directory 
fix https://github.com/chrislusf/seaweedfs/issues/1448
 2020-09-03 19:17:39 -07:00
Chris Lu
b8f32bcab9 filer: compress stored metadata 2020-09-03 11:00:20 -07:00
Chris Lu
f76a2b2c8a printout meta data size 2020-09-03 09:51:21 -07:00
ruitao.liu
798280e980 change filer2 to filer. 2020-09-03 17:05:26 +08:00
ruitao.liu
a93d27d1e8 new filer option to es v7. 2020-09-03 16:34:58 +08:00
Chris Lu
1d56ea24ef fix 2020-09-03 00:08:37 -07:00
Chris Lu
7e1aad0b54 mount: map uid/gid between local and filer 2020-09-03 00:07:22 -07:00
Chris Lu
68e878adb5 fix formatting 2020-09-02 21:42:12 -07:00
Chris Lu
0d99a5da3c c*: support kv operations 2020-09-02 18:39:24 -07:00
Chris Lu
645a4af3db mongodb: support kv operations 2020-09-02 17:19:14 -07:00
Chris Lu
ecaa30c408 better error message 2020-09-02 17:17:44 -07:00
Chris Lu
7c770b727c Update abstract_sql_store_kv.go 2020-09-02 10:13:48 -07:00
Chris Lu
05a6e2dc7f sql kv upsert 2020-09-02 10:12:44 -07:00
Chris Lu
ed62f52452 fix sql insert with duplicated primary key 2020-09-02 10:09:49 -07:00
Chris Lu
9ea290aa12 fix error message 2020-09-01 23:21:41 -07:00
Chris Lu
06a1188267 mysql/postgres: support kv operations 2020-09-01 22:48:36 -07:00
Chris Lu
aa40295f06 leveldb: report not found entry 2020-09-01 22:48:23 -07:00
Chris Lu
87d7312bf6 mysql/postgres: properly report entry not found 2020-09-01 22:47:57 -07:00
Chris Lu
72b0a5f1d1 mysql or postgres: log find error 2020-09-01 22:25:17 -07:00
Chris Lu
2815bbe6c0 go fmt 2020-09-01 21:59:26 -07:00
Chris Lu
37234bf3f8 filer store adds kv support 
can compile now, need to implement those unimplemented
 2020-09-01 21:59:00 -07:00
James Hartig
8e54e34576 volume: Don't unmount before deleting volume in copy 
If we unmount first and then delete, the delete fails because the volume
was unmounted. Delete ends up doing the same thing as the unmount anyways.
 2020-09-01 22:00:07 -04:00
Chris Lu
2b14ae5819 fix tests 2020-09-01 01:33:43 -07:00
Chris Lu
d91ec535b3 fix tests 2020-09-01 01:29:13 -07:00
Chris Lu
eb7929a971 rename filer2 to filer 2020-09-01 00:21:19 -07:00
Chris Lu
38e06d783d volume: check disk space before compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1440
 2020-08-31 18:10:53 -07:00
Chris Lu
dbb10e0f0b testing with more than 30days 2020-08-31 17:59:29 -07:00
Chris Lu
dd176cdb8b avoid tiny step pagination 2020-08-31 11:28:03 -07:00
Chris Lu
97733c3dff typo 2020-08-31 10:49:17 -07:00
Chris Lu
c8398bdb09 adjust metrics 2020-08-31 10:41:05 -07:00
Chris Lu
8a1c8e41b3 simplify if else logic 2020-08-31 10:39:24 -07:00
Chris Lu
f0c89cfacd go fmt 2020-08-31 10:23:31 -07:00
Chris Lu
edb9d65e05
Merge pull request #1431 from kmlebedev/wip-prefix-search 
Wip prefix search
 2020-08-31 10:22:14 -07:00
Konstantin Lebedev
9a195bebfd accurate limit 2020-08-31 22:13:13 +05:00
Konstantin Lebedev
18b98f9747 not convert err to string 2020-08-31 21:55:18 +05:00
Konstantin Lebedev
22f32e75c4 rename 2020-08-31 21:52:05 +05:00
Konstantin Lebedev
a8de332beb fix 2020-08-31 21:45:44 +05:00
Konstantin Lebedev
7f013d77d6 add var UnsupportedListDirectoryPrefixedErr 2020-08-31 21:43:32 +05:00
Konstantin Lebedev
60a86cfe04 add const 2020-08-31 21:35:16 +05:00
Konstantin Lebedev
82ea121d09 rm func 2020-08-31 18:50:06 +05:00
Konstantin Lebedev
2ea638f865 filer store wrapper can implement the logic to filter by prefi 2020-08-31 17:13:56 +05:00
Chris Lu
408e339c53 also delete the manifest chunk itself 2020-08-31 00:16:03 -07:00
Chris Lu
dd52037c27 1.92 2020-08-30 22:56:35 -07:00
Chris Lu
33d8c6c617 change log to warning 2020-08-30 21:01:44 -07:00
Chris Lu
e62d7f221f a bit more logs 2020-08-30 21:00:24 -07:00
Chris Lu
22fe4ae573 mount: fix for UrBackup 2020-08-30 21:00:09 -07:00
Chris Lu
ae84a9ee2e add logs 2020-08-30 20:59:04 -07:00
Chris Lu
a41588279a change log level 5 to 4 2020-08-30 20:12:04 -07:00
Chris Lu
56244fb9a1 fix hard coded host address 2020-08-30 10:23:35 -07:00
Chris Lu
f2a8574448 filer and mount deletion resolves manifest chunks also 2020-08-30 02:07:14 -07:00
Chris Lu
99ecf63276 go fmt 2020-08-29 22:28:33 -07:00
Chris Lu
ef75ce8a34 use UTC for all time related folders 2020-08-29 21:01:14 -07:00
Chris Lu
b69cb74c03 read meta logs by timestamp 
pass in event ts when moving logs
meta aggregator reads in memory logs only
 2020-08-29 17:37:19 -07:00
Chris Lu
063c9ddac5 adjust logs 2020-08-29 11:56:22 -07:00
Chris Lu
ca658a97c5 add signatures to messages to avoid double processing 2020-08-28 23:48:48 -07:00
Chris Lu
63ad1abcce watch: follow changes for a pattern 2020-08-28 19:43:04 -07:00
Chris Lu
b7a654a318 file mime avoid saving application/octet-stream 2020-08-28 19:42:40 -07:00
Chris Lu
707192f966 fix help message 2020-08-27 00:13:38 -07:00
Chris Lu
0be6863c87 rename 2020-08-26 22:40:15 -07:00
Chris Lu
50f7e2e89f add log messages 2020-08-26 09:51:20 -07:00
Chris Lu
6bc618051c log message 2020-08-26 09:32:56 -07:00
Chris Lu
9dae4e9d3e fix logs 2020-08-26 09:31:11 -07:00
Chris Lu
4c1a3187bf fix log error 2020-08-26 09:25:54 -07:00
Chris Lu
ab759f0ec2 erasure coding: fix EC error if multiple disks are configured in one volume server 2020-08-26 09:16:58 -07:00
Chris Lu
d1cf39f180 fix logging 2020-08-26 08:55:15 -07:00
Chris Lu
83012f543a s: avoid possible completeMultipartUpload re-tries that can overwrite the uploaded file 2020-08-26 08:50:24 -07:00
Konstantin Lebedev
464d4c82ec stop send heartbeat before stop volume server 2020-08-26 16:16:11 +05:00
Chris Lu
1901f15cd2 volume server: remove whitelist for status checking 
Volume liveness/readiness not work if jwt enabled, I try use param
-whiteList
but jwt breaks
 2020-08-25 02:27:47 -07:00
Chris Lu
81e5124faf s3: list bucket permission change from admin to read 
fix https://github.com/chrislusf/seaweedfs/issues/1430
 2020-08-24 11:22:45 -07:00
Chris Lu
5f55a87101 close http response 2020-08-24 00:32:44 -07:00
Chris Lu
0b301b0b47 consume all response body in order to release requests 2020-08-23 21:32:29 -07:00
Chris Lu
2cdad77d82 fix checkin error 2020-08-23 18:30:11 -07:00
Chris Lu
81a8aa6581 1.91 2020-08-23 17:28:40 -07:00
Chris Lu
51346a5930 always keep the manifest list of chunks 2020-08-23 17:15:12 -07:00
Chris Lu
c1d1677a28 keep manifest chunks forever 2020-08-23 16:59:01 -07:00
Chris Lu
98175548c2 adjust deletion ordering 2020-08-23 16:27:12 -07:00
Chris Lu
f0e50e7b26 fix test 2020-08-23 16:12:34 -07:00
Chris Lu
aee27ccbe1 multiple fixes 
* adjust isOpen count
* move ContinuousDirtyPages lock to filehandle
* fix problem with MergeIntoVisibles, avoid reusing slices
* let filer delete the garbage
 2020-08-23 15:48:02 -07:00
Chris Lu
c19245886c simpler test 2020-08-23 14:20:27 -07:00
Chris Lu
df816a58fe add tests 2020-08-23 14:09:25 -07:00
Chris Lu
77393d3d30 add file handle locking when changing file entry 2020-08-23 00:35:50 -07:00
Chris Lu
d60bcbf08a sorting chunks 2020-08-23 00:00:36 -07:00
Chris Lu
5e6b714836 add random test 2020-08-22 16:33:00 -07:00
Chris Lu
42ddbcc3a2 SQL insert falls back to update 
fix https://github.com/chrislusf/seaweedfs/issues/1429
 2020-08-20 07:52:46 -07:00
Chris Lu
f48567c5c6 remove unused function 2020-08-19 22:53:49 -07:00
Chris Lu
c45ba5d7d4 fix listObjectsV2 response format 
fix https://github.com/chrislusf/seaweedfs/issues/1426 issue 1
 2020-08-19 12:08:04 -07:00
James Hartig
3ccfa4c6ad Added VolumeMarkWritable and VolumeStatus grpc methods 
This is necessary for copy to mark as read-only and then restore the
original state afterwards.
 2020-08-19 11:42:56 -04:00
Chris Lu
3b4b1d4a77 fix tests 2020-08-19 01:37:56 -07:00
Chris Lu
839634097f also do flush on release 2020-08-19 01:27:10 -07:00
Chris Lu
c27e18aa6a read possible old deleted chunks 2020-08-19 00:43:07 -07:00
Chris Lu
a78772d5ea avoid shutdown in the middle of running 2020-08-19 00:42:02 -07:00
Chris Lu
ae9bc4a508 logs 2020-08-18 23:42:26 -07:00
Chris Lu
ed4b43b419 adjust logs 2020-08-18 23:42:09 -07:00
Chris Lu
9f1e0aeef5 delete chunks in the last step 2020-08-18 23:40:53 -07:00
Chris Lu
99d05f758c adjust logs 2020-08-18 23:39:18 -07:00
Chris Lu
fe01191b5b support read option readDeleted=true 2020-08-18 19:22:16 -07:00
Chris Lu
6ccd7f0a4d refactoring 2020-08-18 18:01:37 -07:00
Chris Lu
c026eb0592 refactoring 2020-08-18 17:39:29 -07:00
Chris Lu
7e91ae592c pass in option to read deleted entries 
not working yet
 2020-08-18 17:37:26 -07:00
Chris Lu
51ecb49db3 for debugging 2020-08-18 17:35:39 -07:00
Chris Lu
ee11d98650 refactoring 2020-08-18 17:35:19 -07:00
Chris Lu
6a92f0bc7a refactoring to typed Size 
Go is amazing with refactoring!
 2020-08-18 17:04:28 -07:00