Commit Graph

8537 Commits

Author SHA1 Message Date
Radtoo
fbb14e0ea8 Adding separate toggle to purge absent vols
While this toggle is basically required to clean out entries for deleted volumes, having a separate description + toggling this separately seems like a good idea so people get a chance to check if their volumes are all mounted/connected as expected.

Also renamed forcePurge to just purge.
2022-02-06 23:46:52 +01:00
Radtoo
6b17f45da2 Also delete paths for which a volume is entirely absent. 2022-02-06 23:22:04 +01:00
Radtoo
fa0cfdfb7f Added basic deletion capabilities to fsck. 2022-02-06 17:05:58 +01:00
chrislu
433fde4b18 move error to a separate file
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
2022-02-04 22:57:51 -08:00
Chris Lu
a23fcb9a7c
Merge pull request #2634 from kmlebedev/errorMetrics
error metrics for filer and store
2022-02-04 22:35:13 -08:00
Chris Lu
247bbabda5
Merge pull request #2632 from lapshin-vitaly/s3api_errors
add s3api error for copy in file, not directory
2022-02-04 22:34:34 -08:00
chrislu
ced3b89395 add util package 2022-02-04 21:34:58 -08:00
chrislu
affe3c2c12 change to util.WriteFile 2022-02-04 21:32:27 -08:00
chrislu
76e297d64f sync call to write file, avoid vif loading error
fix https://github.com/chrislusf/seaweedfs/issues/2633
2022-02-04 11:14:04 -08:00
Chris Lu
d8e6bd0a94
Merge pull request #2635 from zerospiel/rearrange_handlers 2022-02-04 09:40:11 -08:00
zerospiel
f3364fec99 weed/s3api: rearrange s3 methods handlers to ensure correct methods requesting
Otherwise current calls for some methods (i.e. GetObjectAcl) ends up with wrong method selection (i.e. GetObject).

Added generic comment rule of traversing methods
2022-02-04 15:14:48 +03:00
Konstantin Lebedev
9978f54acf fix metric names 2022-02-04 16:45:16 +05:00
Konstantin Lebedev
3f4e17aa24 error metrics for filer and store 2022-02-04 14:07:14 +05:00
Lapshinn Vitaly
6bdc274d4d add s3api error for copy in file, not directory 2022-02-04 03:28:37 +03:00
chrislu
6cf2e7d493 s3: listObjectParts return ErrNoSuchUpload if does not exist
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api abort-multipart-upload --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api list-parts --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
{
    "Initiator": null,
    "Owner": null,
    "StorageClass": "STANDARD"
}

If we abort a multipart upload, it appears that records are left behind. We should get a 404 NoSuchKey error.
2022-02-03 12:34:16 -08:00
Chris Lu
a7982bd525
Merge pull request #2631 from zerospiel/bucket_handlers 2022-02-03 10:17:18 -08:00
zerospiel
b54a65ba5a weed/s3api: added new bucket handlers for more compatibility with AWS S3
Protocol

Otherwise any requests to the underlying handlers results in calls to
ListObjects (v1) that may intensively load gateway and volume servers.

Added the following handlers with default responses:
- GetBucketLocation
- GetBucketRequestPayment

Added the following handlers with NotFound and NotImplemented responses:
- PutBucketAcl
- GetBucketPolicy
- PutBucketPolicy
- DeleteBucketPolicy
- GetBucketCors
- PutBucketCors
- DeleteBucketCors
2022-02-03 17:17:05 +03:00
chrislu
6bee1e9714 [volume.check.disk] fix wrong logic to compare
fix https://github.com/chrislusf/seaweedfs/issues/2626

minuend - subtrahend
2022-02-01 07:48:28 -08:00
chrislu
7270067289 2.88 2022-01-30 20:25:26 -08:00
chrislu
84c9bc4389 edge case: old entry was not replicated to remote storage 2022-01-30 20:23:24 -08:00
chrislu
b8490fe427 adjust volume count even when not applying the changes 2022-01-28 19:11:46 -08:00
Chris Lu
b3f0f170b4
Merge pull request #2618 from divanikus/master 2022-01-28 03:34:52 -08:00
chrislu
b9b684194f remove max connection age
following https://github.com/grpc/grpc-go/issues/3170#issuecomment-552517779
2022-01-27 02:28:22 -08:00
divanikus
67e3fe996a async volumeTierMove 2022-01-26 18:22:31 +03:00
chrislu
62d815d1ca use limited in memory buffer instead of swap file 2022-01-26 00:36:19 -08:00
Chris Lu
9596fce562
Merge pull request #2613 from kmlebedev/masterPromMetrics
Master prom metrics
2022-01-25 08:13:14 -08:00
Konstantin Lebedev
5c9259fa3c fix metrics master name 2022-01-25 14:42:47 +05:00
chrislu
7328962009 revert "POSIX: should not delete if a directory is not empty"
revert 0c75f15062

Reported:

Hi, about commit: 0c75f15062
POSIX: should not delete if a directory is not empty
It should still delete with a command like rm -rf "${path}"/  because it is a forced delete, but now it gets fail to delete non-empty folder: [...]
Can you enable the delete if it is forced?
2022-01-24 14:21:50 -08:00
Konstantin Lebedev
c9952759c4 metrics master is leader 2022-01-24 20:13:07 +05:00
Konstantin Lebedev
28efe31524 new master metrics 2022-01-24 19:09:43 +05:00
chrislu
3bba2124ef use a sliding window of in-memory writable chunks 2022-01-23 23:02:05 -08:00
chrislu
520591e6ea reset swap file chunk after uploading 2022-01-23 18:30:53 -08:00
chrislu
e185d90d24 2.87 2022-01-23 16:18:55 -08:00
chrislu
4a311c7f5e dedup local metadata subscribers
fix https://github.com/chrislusf/seaweedfs/discussions/2542
2022-01-23 16:14:22 -08:00
chrislu
bb6854b972 adjust log 2022-01-22 12:35:09 -08:00
chrislu
2bdd737971 rename file 2022-01-22 08:28:35 -08:00
chrislu
643bbbeb49 rename 2022-01-22 08:27:40 -08:00
chrislu
18ed06b420 use memory when under 16 chunks 2022-01-22 08:11:01 -08:00
chrislu
8e80f3cd65 move upload pipeline locking to a different file 2022-01-22 08:09:55 -08:00
chrislu
dde34fa99d rename 2022-01-22 08:08:01 -08:00
chrislu
a2aa542370 rename 2022-01-22 08:07:11 -08:00
chrislu
c376ccc5a5 swap file based random write large file upload 2022-01-22 08:06:53 -08:00
chrislu
b9ae16fbc5 fix memory allocation 2022-01-22 08:05:04 -08:00
chrislu
1ee828b768 refactor
do not expose internal offset
2022-01-22 06:34:29 -08:00
chrislu
02d0c12cdd rename 2022-01-22 06:00:10 -08:00
chrislu
5dea5c0449 refactor 2022-01-22 05:59:07 -08:00
chrislu
8aa6bf0bb9 refactoring 2022-01-22 05:40:10 -08:00
chrislu
d97bd54e63 just refactoring 2022-01-22 04:18:54 -08:00
chrislu
482014f9da rename file 2022-01-22 03:50:18 -08:00
chrislu
3b4a9addaf rename 2022-01-22 01:46:10 -08:00